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ABSTRACT 



A summary of the first 18 months of operation of the 
Rocky Mountain States' Educational Technology 
(renamed Satellite Technology Demonstration after Bay 
the history of the demonstration and explains the 
the demonstration's evaluation planning and historical 
report concludes with recommendations for future social 
projects, noting that it is inherently difficult to 
achieve a workable balance among political constraints, operational 
constraints, social constraints, and at the same time reach the 
project's goals, (DGC) 



Federation of 
Demonstration 
1973) details 
rationale for 
analysis. The 
demonstration 



ERIC 



o 



Final Report 



Contract No. nnW-OS-72-155 , L. 
Principal Investigators 



M. Nelson and N, 



Maccoby , 



Nancy H. Markle 
David G. Markle 
Conrad G. Carlberg 
Dennis R. Foote 

Stanford University 
Department of Communication 
Stanford, California 94305 



HISTORY AND RECOMMENDATIONS RESULTING FROM 
EVALUATION PLANNING FOR THE FEDERATION OF 
ROCKY MOUNTAIN STATES' EDUCATIONAL TECHNOLOGY 
DEMONSTRATION 

Edited by Nancy H. Markle and David G. Markle 



ERIC 



0 

0 

M 



10 Mav 1974 



O \ OIPA»TMENTO» HEALTH 
EOUCATIOM A WE AVE 
NATIONAL INSTITUTE OP 
COUCATION 

• , VI ' HI » 

u (- "N ;w •< 

• , I , f A W t' OS 



U. S. DEPARTMENT OF HEALTH, EDUCATION, AND WELFARE 

Office of the Secretary 



I 



Final Report 



Contract No. HEW-OS- 72- 1 55 
. M. Nelson and N. Maccoby, Principal Investigators 



HISTORY AND RECOMMENDATIONS RESIILTINC PROM 
EVALUATION PLANNING FOR THE 
FEDERATION OF ROCKY MOUNTAIN STATES' 
EDUCATIONAL TECHNOLOGY DEMONSTRATION 



Nancy H. Markle 
David G. Markle 
Conrad G. Carlberg 
Dennis R. Foote 

Stanford University 
Department of Communication 
Stanford, California 94305 



10 May 1974 



The research reported herein was performed pursuant to a 
contract with the Office of Education, U.S. Department of 
Health, Education, and Welfare. Contractors undertaking 
such projects under Government sponsorship are encouraged 
to express freely their professional judgment in the con- 
duct of the project. Points of view or opinions stated 
do not, therefore, necessarily represent official Depart- 
ment of Health, Education, and Welfare nosition or oninion. 



U.S. nr.PARTMHNT OF HEALTH, fiDUCATION, AND WELFARE 
Office of the Secretary 



Abstract 



Purpose: This report gives the history of approximately 
tTie fi T st two years of the satellite telecommunications 
demonstration project beii-g planned by the Federation of 
Rockv Mountain States in Denver, Colorado. In addition 
to history, the report summarizes the evaluation planning 
activities and provides recommendations for planning future 
proj ccts. 

Methodology: In carrying out tha Contract and in writing 
this report our* roles and methods combine these ot parti- 
cipant observer, interviewer, and researcher of documents. 

Results and Conclusions : This report is prepared in five 
chapters. They are: T. Introduction: Participants and 
Setting in the Harly Planning Phases; II. 

Evaluation Planning, Rationale for Format and «J ^^^/^.^.f .^^^ 
Analysis; III. Narrative Chronological History; ly. History 
from the I:TD Component Function Viewpoint; V. Analysis o. 
the History of the Federation of Rocky Mountain States 
Educational Technology Demonstration and Recommendations 
for Future Projects. 



*Sranfora University, Department of Communication Field 1 
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Preface 



"he nurnose of this Final Report is to summarize the 
first eip.hteen months of operation of the Federation of Rocky 
Mountain States' F.ducational Technology ^lemons trat ion (renamed 
after Mav 1973: Satellite Technology Demonstration) and to 
provide an analysis of the events the project that will 
be Mseful to government officials, students of public policy, 
and others who may be interested. 

Most research is either evaluative, diagnostic, or pre- 
scriptive. SoTne programs involve aspects of all three. 
Fvaluative rtsearch is the easiest to perform, and prescrip- 
tive (since it involves prediction) is the most difficult. 
Although the present effort deviates somewhat from the usual 
conception of research, we have made an attempt to combine 
the three of the types, evaluative, diagnostic, and prescrip- 
tive, into a meaningful whole. In doing this we have used 
participation, observation, and historical recording and 
analysis techniques. It is hoped th.it recommendations derived 
from this project will be generalizable to similar undertakings 
and to some of the ever more frequently occuring cooperative 
arrangements being developed by various levels of government 
and local communities on numerous projects. 
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('haptcr T 

TN'TRnnilCTION: nARTTCIPANTS AND 
SHTTINT, IN THI; F.ARLY MANNING PHASP.S 

Nancv H. Markle» Havid H. Markle, Conrad C. Cnrlhcrf 



Parly History 

In 1971 planners in the National Aeronautics and Space 
Administration CNASA) , the Department of Health, Pducation, 
and Welfare (ilPW) , and the Corporation for Public Broadcasting 
(CPB) , were seeking organizat i ons to carry out demonstrations 
usinti the Applied Technology Satellite-P, which would he 
geostationary over the Rocky Mountain area for the first vear 
after its launch--then to be early in 1973- - (71/06/02. 71/n6/2A, 
71/06/02R, 71/06/14A, 71/06/14B).i 

The Poderation of Rocky ^'ountain States was aware of 
potentials of satellite communications and hepan to discuss 
mutual interests with NASA and HP.W (72/ir-/04A. 71/01/14, 
72/11/16). In May of 1971, FRMS received a preplanninp contract 
and began to work on what would be needed to plan a satellite- 
assisted demonstration for the delivery of social and educa- 
tional services for the region (72/02/24). The services of 
the system were to be based on perceived needs and wants of 
the potential system users. 

Pour organizations with regional interests were to coonerate 
with HHW and NASA to produce the Demonstration: PRMS, the 
Pducation Commission of the States (PCS), the Western Interstate 
Commission for Higher Pducation (IVICKP) , and the ^ocky Mountain 
Corporation for Public Broadcasting (RMCPR) . 



The Federation of Pockv Mountain States 



'he Federation of Pocky Mountain States is a non-profit, 
ro^ionallv- based organization. Its member.^ include the 
(;OVcrnors of six states (Idaho, Montana, Utah, Wvomirv, 
'uolorado, and New Mexico), as well as many professional anJ. 
educational associations, and cor.^orat ions . The PRMS offices 
are in Denver, rolorado. The President of the Federation is a 
former Covcrnor of Vow ^^oxico. The Federation eneares in 
a number of activities in\:cnded to promote b-jsiness and 

v. T^i f- n ur^HM* s "r o f c r to docur.ents, by dnto, that were sources 
for information. The documents are listed in the nci'orQnco<, , 
For exannle, ■'1/06/02A indicates the vear f]071), the nonth 
(06 - - Ttme) , the dnv (?.) , atid that it is the "\" document 
in a rroun with that date. 
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inchistrv, t ransnortnt ion , land use, cducatioi, performing 
arts, ami .so forth, for the six-member-state repion. Such 
an association is notontiallv beneficial, because the RocVv 
Mountain reRion has a hi^h proportion of land to inhabitants, 
and bv sharinp, or in soine cases, by centralizinp services, 
each state may deiive ^ireater benefits from existing and 
potential resources. 

The Federation is a relatively younp organization, 
founded in 1066. The HTD is its first larpe pro.iect. Tt 
seemed to the original planners of the Satellite prorrams 
that the Federation was a logical choice to centralize the 
satellite- related activities for the region. Originally, 
the Federation would puide the project and provide a small 
career-oriented propram and utilization services, '^he 
fducation rnmmission of the Stptes (FCS) would, under sub- 
contract, provide planninc and manacjement services, as well 
as nrovide content for ea.*ly childhood education pro- 
gramminp. The Western Interstate Tommission for Higher 
F.ducation (WICIIFO would provide programming in the Higher 
r.ducation field. The Cornoration for Public Broadcasting 
would provide advice, some facilities, air time, and 
services. The Rocky Mountain Projects would cooperate 
and share ideas and some equipment with educational and 
social services projects in Alaska and in the Appalachian 
Region . 

"^hc first planning proposal was submitted in Julv 19'^1 
("2/11/16). Three content area? with high national and 
ret;ional priority were identified: early childhood development, 
career development, and higher education. 

After revisions, made for the purpose of limiting the 
scope 0^^ the proposed activities, a planning contract was 
awarded on the basis of negotiations conducted in Tnnuarv 
19^2 (7:/01/ni. "^his contract (later amended to bccoine a 
grant! provided Sson^noo from the Office of F.ducation. 

Thoro have been nanv developments since the tine o*" that 
^■irst rrant. These rue discussed in subsequent chanters of 
the present ^inal "eport. 



t'Jucation rommission o^ the ^tates 

The r-lucation rommission the *^rates is also a rela- 
tive Iv voiinr organization, founded in 19f''">. ?CS ha'^ its 
hea('n>iartors in I^enver, but it has a broader national hase 
than I-RM^. At the tine of the ("Tn Proiect, VCS claimed the 
rovornor.^ o*" i.: states a^ npnhor^;. The largest rr^ nrnioct 
to date is the National Assessment, which it inher i ted fror. 
t-hc ^'onnittee for Assessing the Progress rducation in 
IQ"!. res has not been involved with the liTH since *'av ni'!^. 
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Western Interstate Commission for Higher T-ducation 

WICin- is a non-profit agency created bv 13 western states 
(Alaska, Arizona, California, Colorado, Hawaii, Idaho, Montana, 
Nevada, New ^1exico, Oregon, Utah, Washington, and Wyoming). The 
governor of each participating state appoints 3 Commissioners 
to the Coverning Board of the Commission. The Commission 
administers the Western Regional Pducation Compact, which is 
an agreement among the states to work cooperatively to improve 
educational programs. WIC!IE was formally created in 19 SI; 
nrogram activities began in 1953. The Commission terminated 
its involvement with the activities of the F.TD, after the 
funding was reduced in 1973. 

Rocky Mountain Corporation for Public Broadcasting 

Ve Rocky Mountain Corporation for Public Broadcasting 
is asso:iated with the nationally-based Corporation for 
Public . roadcasting. As stated earlier, members of R^^CPB 
were to provide advice, some facilities, services, and air 
time for the Project. However, it has been decided more 
recently to locate the satellite projects* uplink to the 
ATS-F in FRMS - operated premises in nenver and to establish a 
small production studio also in FRMS operated premises. \s 
a result, the potential role of RMCPB is not as clearlv 
defined as it might have been under other circumstances. 

Related Projects 

The related Alaskan and Appalachian satellite projects 
were somewhat slower in becoming operational than the FR^'S 
FTP. In addition, these projects were somewhat less 
ambitious in initial conception than those of the FRMS F.TFi. 
Both are presently active, with some aid in engineering and 
equipment from FRMS. 

Historical Context for the Demonstration 

The present demonstration is an ongoing effort in 
the context of changing federal policy, with an increase in 
state participation and involvement in national and regional 
decision making, nne element of this shift in national polic' 
is an Administration effort to reduce tho amount of Icnislation 
intended to achieve very specific purposes in American Ofluca- 
tion (catecorical grants), and to increase broader revenue 
sharinc rrants. The implementation of this shift in nnlicv 
is an important part of the h-storical context of the '^emon- 
*;tration because associated federal decision making hari an 
influence unon federal planning for the project, allocation of 
funds, and monitoring arrangements. 



The project was funded and began operations durinp a 
Tjeriod of several national debates on educational policy 
and the role of the federal pcernment in education. 
For example, as a result of nroposals presented by the Depart- 
inent of Health, Hducation, and Welfare the role of the federal 
.government in higher education was being debated in Congress. 

'Educational renewal** w.'.s also being discussed within 
Congress and the Administration. The basic idea was to focus 
federal discretionary monies on the areas of highest need in 
the United States, primarily rural areas and major cities. 

Legislation for the National Institute of Hducation, which 
had originally been proposed several years earlier bv an advisor 
to the President was also being considered in Congress. The 
Office of nducation was in the nrocess of a reorganization. 
The cummulative effect of these events was an atmosnhere of 
continuous change in the formulation and administration of 
federal education policy in Washington. 

We cannot fully interpret the effects of these events 
unon the Demonstration because complete documentation is not 
available. The project has been affected by debates on 
educational renewal in NIE and the Fund for Pos tsecondary 
Fducation, and by administrative/legislative negotiations on 
revenue sharing. Written public records of these negotiations 
are limited. There is no written record of numerous discussions 
concerning the nemonstration- -discussions about the role of 
federal-state cooperation, levels of federal funding, and 
of negotiations between the states. Consequently, some parts 
of the history have been based on extrapolations from existing 
data and observed changes in the operation of the nroiect. 



The Stanford University Department of Communication 
Field Team 

In Mav of 1972, the Office of the Secretary, ^^eDartmont 
0^ Health, r.ducation, and Welfare awarded an 8 month contrnct, 
later extended to an 17 nonth contract ending 8 October 19"3, 
to the .Stanford University Dopartment of Communication. The 
nurposo of the contract was to provide an adiunct evaluation 
nlanning staff for the FRMS liTD (72/04/07) (7:/05/nS). 

The Stanford contract called for the performance of six 
major tasks. These tasks were completed at the interim re- 
nortine period (8 Tanuary 19 7.-5). Two tasks (1) and (2) wore 
also continued by the contract extension until 8 October 1373. 
These tasks, and' their resultant documents were: 

Task 1: Provide advice and guidance on evaluation to 
the nemonstration planners during the design of the 
experiment. 
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Task 2: Provide an analysis of the historical record of 
the Demonstration planning process. 

Markle, Nancy H. "Ceneral History and Analysis 
of the Planning Phase of the Federation of Pockv 
States Educational Technology Demonstration." 
Part IV. Interim Report, Contract No. HPW-os-72- 
155, Evaluation Planning for the Rocky Mountain 
Demonstration in Pducational Technology , TT^ 
Nelson and n. Maccoby, Principal investigators , 
Stanford University, Stanford, California. 
19 January 1973. 

Hall, Douglas ^. "The Historical Record of the 
Planning Phase of the Federation of Rocky Mountain 
States Educational Technology Demonstration." Part 
VIII. Interim Report, Contract No. HHIV-OS- 72- 155, 
P.valuatic- - Planning for the Rocky ^Tountain demo n- 
stration 1.;' Educational Technology , L.M. Nelson'" 
and \'. Maccoby, Principal Investigators, Stanford 
Hniversity, Stanford, California. 19 January 1973. 

Task 3: Provide a statem<=^nt of the educational obiec- 
tives for each segment of the Early Childhood and Career 
Education components of the Demonstration. 

FRMS ETD Career Development Component. "Objectives 
Prepared by the Content Components of the Educational 
Technology Demonstration." Part III A. Interim 
Report, Contract No. HEW-nS- 72- 1 55 , Evaluation 
Planning for the Rocky >!ountain demonstration in 
Educational Technology , L.M. Nelson and N. ^c c o b v , 
Principal Investigators, Stanford University, 
Stanford, California. 19 Januar>' 1973. 

FRMS ETD P.arly Childhood Development Component. 
"Objectives Prepared by the Content Component of 
the Educationai Technology Demonstration." Part 
III B. Interim Report, Contract No. HEW-OS-72-1S5, 
Fvaluation Planning for the ^ocky Mountain Dei.ion - 
stration in Educational Technology , L.^-t. Nelson 
and N. Maccoby, Principal Investigators, Stanford 
University, Stanford, California. 19 January 1P"3. 

Task 4: Provide a specific plan, including dra^t 
instruments, for documenting what was done during the 
Demonstration. 

Foote, Dennis P. "A Draft Documentation Svstem 
for the liducat ional Trchnology 'Vmons t rat ion . " 
Part II. Interim Renort, Contract No. HEW-'^S-TZ- 
15 5, Fvaluation Planning for the Rockv Mountain 



Demons t rat ion in Educational Technology , L.M. 
Nelson anci f^. Maccohy, Principa l investigators , 
Stanford University, Stanford, California. 
19 January 1973. 

Task 5: Provide a specific plan for formative 
evaluation to be made during the earlv stages of the 
Demonstration, and for the introduction of imDrovemcnts 
during its progress. 

Marklc, D.n. "Interim Planning for Formative 
Evaluation of the Fducational Technology Demon- 
stration." Part I. Interim Report, Contract \''">. 



Mountain Region Demonstration in t-ducational 
Technology , L.M. Nelson and N. Maccohy, Principal 
Investigators , Stanford University, Stanford, 
California. 19 January 1973. 

Task 6: Assess the problems and possibilities involved 
in making a summative evaluation of the Demonstration. 

Markle, Nancy H. "Problems and Possibilities 
Involved in Making a Summative F.valuation." 
Part V'. Interim Report, Contract No. I^FW-ns- 
72- 155 , Evaluation Planning for the gf^cky ^Jountain 
Region Demonstration in r.ducational^echnologv , L.M 
Nelson and N. Maccohy, Principal Investigators, 
Stanford University, Stanford, California. 
19 ./anuary 1973. 

An additional product of the first period of contract work 
was a paper on a method for cost/accounting: 



^larkle, D.C., N.H. Markle, and nennis P. Foote, "An 
Approach to Cost/Benefit Analysis, Federation of 
Rocky Mountain States' Fducational Technology 
Demonstration." Used for Examples. Technical 
Renort, Contract No. HF.W-OS- 72- 155 , Fvaluation 
Planning for the Rocky Mountain Region nemon sTrat ion 
in Educational Technology , L.M. Nelson and \. 
^^iccobv. Principal Investigators, Stanford Univer- 
sity, Stanford, California. U.S. Department of 
Health, Education, and Welfare, office of the 
Secretary, 6 July 1973. 



As mentioned above, in January of 1973 the original 
contract was extended for 9 months to continue with Tasks 
1 and 2 and to be.c:in implementation of some of the recom- 
mendations in Task 5. As a result of changes in the f'TD in 




May 1973, both in fundinp and the resulting project organi- 
zation and objectives, products of Tasks 3, 4, and 5 are not 
being implemented in the present demonstration. 2 



Background for the T-RMS T-TD 

The rationale for an Educational Technolopy Demonstration 
was based on an analysis of numerous issues in education and 
technology in the late 1960s and early 1970s, One aspect of 
the rationale was that the cost of education has been risinp, 
but the returns for the investment in education have not been 
rising proportionally. 

It was argued that the Department of Health, Fducation, 
and Welfare needed better information to guide policv and 
resource allocation decisions, particularly in the field of 
education. This meant that more complete information was 
needed on the costs and benefits of investment in human- inten- 
sive and technology-intensive educational systems. Some of 
this information might be provided from the results of an 
experimental demonstration using frequencies allocated for 
social purposes on Applied Technology Satellite-F. 

These issues were the basis for the creation of the 
nducational Technology Demonstration, even though federal, 
regional, and local officials perceived them differently, 
and with greater or less enthusiasm. Investment in human 
services, technical systems, and software would be measured 
in this Demonstration, and their effectiveness determined. 
The findings might, then, contribute to the improvement of^ 
public services by providing a source of information to guide 
future decisions. 

Manv innovative ideas connected with experimental educa- 
tional and service delivery programs were part of the discus- 
sions about an experimental demonstration. It was natural 
enough for the people involved on-site to become committed to 
answering as many questions as nossible and to providing as 
extensive a set of services as possible. Since federal 
officials saw the demonstration less broadly, this was one 
basis for a mismatch in expectations between federal and on-si 
planners that plagued the Demonstration throughout its 
entire preplanning, planning, and early operations periods. 



T. The reports that resulted from this contract are available 
i:r>on request from: Stanford Dnivcrsity, Department of Tommuni 
cation, Stanford, California 94305. 



^n*]^ T^lannpd to design educational services that would he 
responsive to the recipients' needs and desires. The Hemon- 
stration was to show that non- contiguous communities of 
interest could he served using technology, and that public 
acceptance of technology would he increased. Further, this was 
to be done in a framework that would allow new or expanded 
cooperative arrangements to develop among institutions. These 
institutions could use information from the Pemonstration for 
policy and program decisions. 

The nducational Technology Demonstration planned to eather 
information about the effect of programming on part icinants , 
the cost of elements of the system, and the degree of public 
acceptance of the system. Such data, it was argued, could 
be used to make cost benefit and cost effectiveness projections 
for future systems that were more technology- intensive than 
existing systems. It was also anticipated that the project 
could provide the basis for advances in research and applications 
of engineering technology; provide information to the scientific 
community about how learning happens in both closed and open 
environments; provide a body of programs that could be used on 
an ongoing basis by institutions in the region; and provide 
information that could he applied over numerous Program areas, 

^nce this information was assembled, existing institutions 
might be able to use it in making decisions about how to improve 
the quality of existing services, how to provide services to 
those who presently lacked them, and how to provide people with 
a wider range of program choices. These institutional decisions 
would lead to cooperative arrangements among agencies at all 
levels of government, public agencies and private agencies. 
Public issues could thus be addressed more effectively than had 
been the case up to the early 1970s. 

Reasonable, relevant national legislation might then 
be forthcoming, allocating broadcast frequency channels for 
use in the public sector. N'ew investment opportunities in 
the telecommunications field could be stimulated and the 
directions of private investment in that industry could bo 
influenced. Eventually, there might be a substantial increase 
in the sharing of resources, a reduction of pressure on public 
funds, more effective methodologies for program development, 
increased productivity in social services, and economies of 
scale achieved from the pooling of buying power. 

This set of goals, while commendable, was extraordinarily 
comprehensive to be accomplished by the one project. These 
goals were never agreed to by all of the participants either 
in Denver, the states, or Washington. The federal participants 
haJ considerable prior experience upon which to base their 
reservations. There were numerous debates on parts, or all, 
of this rationale throughout the early stages of the nroiect. 
The result was continuing discussions of goals, objectives, 
operations, and funding levels for the proiect because parti- 
cipants continued to maintain incongruent exnectat ions . 



status of the Present Report 

The proiect is In operation at the time of this writinR, 
Consequently, the final record is not complete. Portions of 
the Analvsls and Recommendations that refer to the ptT) are 
based mostlv on operations up to the end of September, 1P73. 
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Aoproach to livaluation Planning 



The Stanford evaluation planning efforts were intended 
to be larpolv formative rather than summative in nature.-" 
This section, adapted from an Appendix to an early Stanford 
Progress Report, describes the way that "evaluation" was 
viewed and approached. 



Kin< :s and Purposes of I'.valuation 

The ultimate purpose of evaluation is to supply infor- 
mation that will aid decision-making. There are a number 
of kinds of evaluation, and there are uses for each. 

Cronbach (1963) distinguishes between evaluation o^ 
the results of a course and evaluation for course imnrovemcnt. 
More recently, Scriven (1967) has introduced the terms 
•summative' and 'formative', which cover broader activities 
but correspond approximately with the kinds of purposes 
described by Cronbach. 

\ nlan for the evaluation of the total effort of a project 
will include several kinds of evaluation. Some of these will 
be formative in application and some will be summative. 
r.ach kind may take place to some extent during the planning, 
development, and implementation phases of the proiect. 
Several kinds of evaluative purposes are given below. 

F ormative purposes . 

1; Fvaluation of the choices among alternatives, so 
that both rational and data- supported progress may be made 
in planning, development, and implementation. 

T, 'nri'jr'app roach was not particularly successful with the 
FRMS FTP, since the FTP staff resisted the attempts of the 
'•outside' team, tending to regard formative activities as 
"telling then what to do." For example, none of the ^-^tanford 
team was invited to or allowed to be present at the weekly 
FTD Component Pi rectors Meetings until the end of April IP"."?, 
.ifter the NIF site team visit. 
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2) !-valuation for course improvemont or user services 
improvement (empirical development nrocejjs), which is similar 
in kind to (1), but which deals with products, courses, utili- 
zation systems and with persons, rather than with project 
processes. 

Summative Purposes . 

1) Rvaluation of how well the plannini? served the 
development and implementation. 

2) rvaluation of how well the development served the 
inn lementat ion. 

y>) Evaluation of how well the implementation served 
the Roals and objectives of the nroject. 



Comnarison and Contrast 

Hvaluative information is examined both durinp anc! after 
the project. From it, decisions can be made both about future 
activities within the project itself and about directions to 
take and to avoid in similar future projects. 

Most evaluative activities may have either formative 
or summative applications. Astin and T^anos (1971), in their 
article about evaluation of educational programs, comment that, 
"the basic conceptual and methodological issues appear to be 
equally relevant to problems of 'formative' evaluation." 

The distinction between 'formative' and 'summative' 
evaluation is more, however, than a play on words or ten.noral 
activities. There is a real difference between the two 
activities in purpose, and hence there is often a difference 
in the kind of data collected, the way in which it is collected, 
and the wav in which it is examined. There is, for examnle, 
a contrast of purpose between collecting data to be used for 
strtistical testing and collecting data for svstem develonnent 
and improvement. Furthernore, data may be collected either 
in a formal, rigorous way, or in an informal, casual way. 



Collecting Formative and Summative Hata 

F:vidence should be collected, during both the develop- 
ment and the implementation periods of a project, that 
contributes to decision making or that documents the decision 
makinc: process. The evidence can be obtained in a variety 
of ways: either by formal, experimental design, bv <^nnc 
quas i -experimental design, by recording casual and anecdotal 
data during tryouts of materials, equipment, or personnel 
patterns, or by some combination of these. 



The .liffpi'f^nt purposes of evaluation should be roco^nir.cd. 
Uses of ilata Vot crcntion and production noed not be confused 
with uses of da^o for making inferences or the basis of 
statistical tests. 



rornntive PaM 

The type of evaluation discussed by most writers makes 
use of rigorously collected formal data exclusively, tvhether 
the purpose is fbrrative or sumtaative in nature. The formative 
kind of casually-cfjU ected informal data, collected to 
support decision making in system, product, or course develop- 
ment and improvement, is also useful. Casual data may he 
anecdotal or it may be from a closely-monitored onc-of-a- 
kind performance of a single student, or of several students. 
These data are used as a basis for making chanp.es in systems, 
materials, or methods, prior to a subsequent tryout. Such 
nonrigorous data are usually useless for statistical testing 
of generality of effects. 



Summative Dat a 

Lord and N'ovick (1968) suggest a vay to formulate 
broad coverage instruments from a large population of 
measuring items. They present the methodology wherebv all 
items can be used, in various instruments, if such coverage 
is deemed necessary. 

^fethods for obtaining comparisons for effectiveness and 
cost/effectiveness might be any variation of the following: 

a) Using a single group as its own control, in a sort 
of before-after comparison. 

b) Using a single group as its own control, bv rmnloving 
concomitant variation among suh-groups within the groun. 

c) Using nn outside audience for comparison, via norred 
and standardized tests, for example. 

d) Using the group or sub-group as a sampling unit for 
some analyses- -by this process eliminating a large, exncctcd 
within-group variance in trade-off for a greatly reduced 
number of degress of freedom. 

A variety of designs for experimental data collection 
and suggested associated analyses may he found in Rloom, 
et al, rnmpbell and Stanley, 19<>3; Wnrthon an.' ^mdfrs, 

T?7T7 and Winer, lOr-.:. 



I'onclu? ioT; 



Therr should be no competition betwpen the purposes of 
summative and forira?: i v<» evaluation. Both are necessary. 
The activities complement one another. The kind;.' of data 
that are collected in any riven case should be deternitied 
the questions to be answered ny the invest iRation» as wpII 
as bv the quality of data that are obtainable and the nossible 
conditions of aditiinistration. 



Rationale for Format of The TTH Historical 
Record and Analysis 



Huring staff discussions of how to approach rhe Historical 
Record and An?lysis it was determined that there are at least 
five possible ways to approach a historical analysis, each with 
certain advantages and disadvantages: 

1) Strictly chronological: although this approach 
is minimally subiect to author bias, it does not encouraj^e 
the raising' of issues that may be both pertinent and difficult 
to document when available record;; do not provide sufficiently 
detailed information. Further, a strictly chronological 
approach is likelv to prove uninformat ive to certain agencies 
interested in the' history of the T-TD, e.g., HHW, nr.. Nil*. 

2) Organized around particular project tonics or 
functions: this approach obviates the fragmentary nature 
of a strictly chronological format, in that it allows for 
the inclusion of -^ertain information, not immediately 
available in document form, as context. It will, howevf»r, 
tend to de-emphasize imnortant problems that do not nortain 
exclusively to one topic or function. The tonic or *"unction 
approach is subject to author bias, and may well organize 
information in a manner not useful to other interested 
agencies . 

3) Organized around the structure of the r.TP: this 
approach will very likely involve the discussion of specific 
persons more than is necessary or desirable, and it is 
questionable general i zab i 1 ity to other projects. It mav 
also de-emphasize issues that apply more to the project 

as a whole than to specific structures within the project. 
While this particular structure is arbitrary, it is not 
immune to author bias. 

1) Prohlen-oriented approach: this approach will 
appear more negative than any of the others available, nav 
appear fracmented, mav neglect some of the available infor- 
mation, and is based on the assumption that certain sensitive 
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infrrnfition can hr fiathcrci'. However, a comb ln*it ion of the 
topic-orionted approach and the prcbletn-oricmted approach 
appear likely to provide infnrmation of use to other a«?encic» 
and future planners. 

We h?ve dec^.ded to combine several of these approaches 
in different sections. A disadvantage of this combination 
is that the reports are somewhat redundant. An advantape 
is that tho history is covered from a variety of viewpoints. 



Tne Historic Document Pile for the 
Federation of Rocky ^fountain vStatcs' 
F.ducational TechnolOiiy ncmonstration 



One of the continuing tasks for the Stanford Field Team 
was to consult project documents for history recording nurnoses 
and to maintain a file of historic documents for the Project. 
These activities were carried out with the four following 
data sets: 

1) Stanford rvaluntion Planning Field Team files containing 
copies of important Planning and negotiation documents 
obtained during the course of field team activities 

or supplied by the contract monitor and other Federal 
persons . 

2) The personal records of the then Assistant Director of 
the project (through Anril 1973) and the personal files 
of the Project's Research Director (at that time). 

liach of these files contains several thousand documents-- 
many of which are duplicated in the Stanford Field Team 
Document files. 

3) The project files of Federal nepar:nent and Agencv 
personnel, some of which are duplicated in the Stanford 
Field Team Document files. 

4) N'ot all of the FRMS FTP Directors and Personnel saw fit 
to provide the Stanford Field Team with copies of 
nlannine and negotiation documents cr records, '■f^'orts 
of the Production, Utilization, and f'areer ^evelomnent 
comnonents to be helpful make our records of day to day 
activities for those areas nore complete than for other 
areas . 

These files contf'in nearly 5 , 000 documents, coded on the face 
of the document into t.ie following categories: 
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I. Structure 

A. Mananement Structure 

B. nCS-TRMS relationships 

C. f'TD relationships with other agencies 
T>. Personnel 

r,. PMPT 

F. Consultants 

II. Oecision-makinR 

\. Project scope 

B. Internal priorities 

C. Project output versus constraints 
n. Hecision identification 

III. Tasks 

A. Site selection 

B. Funding and Budgeting 

C. Production 

P. Programming content 

P.. Research 

F. F.valuation 

n. Hardware and systems design 



The most significant of thise documents were used in the 
preparation of this Final Report. Pay to day correspondence, 
memos, and other documents that were not representative of 
significant Project or Federal activities were not coded for 
the historic document file. 



Request for Project Information 



As the Stanford involvement with the nTPi drew to a 
close, a request for information was sent to the senior 
nersons who had been involved up to that tine. The rollnwinc 
facsinile details the contents and recipients of the letter. 
The responses received from FTP personnel are in the Nnnenflix. 



STANFORD UNIVERSITY 

STANFORD. CALIFORNIA 94305 



OrPArtMINt HI COMMfNICAriON 



llor KY MOl'KTAIN PmtjH^T OfVU.U 
DtMnt. tXM«J« Mail 

(101) 4ta'ifii 



7 August 1973 



novernor .Tack M. Campbell 

President, Federation of Rocky Mountain States 

Suite 300 B 

2480 W. 26th Avenue 

Denver, Colorado 80211 

Dear Ciovernor Campbell: 

We have mailed a copy of the letter below to the 
following list of persons. If you can think of anyone 
else who should, perhaps, receive such an invitation to 
respond, will you please let me know? 

Thank you very much. 



Mr. Michael H. Annison 

nr. Louis Bransford 

Dr. John Cameron 

Mr. Fred P.hrahimi 

Dr. Larry Grayson 

nr. Albert Horley 

Mr. Gene Linder 

Dr. Richard Marsten 

Mr. Arthur Melmed 

Dr. James Peterson 

Mr. Al l^alen 



Dr. David Berknan 

^Ir. Jerry Brasher 

I^'s. Pam Cough 1 in 

Dr. Robert Filep 

Dr. ndith Hrotberg 

Dr. Cordon Law 

Dr. Kenneth Lokey 

Mr. Ben Mason 

Mr. Dail Dgden 

Dr. Alice Scates 



"As you know, Stanford University, Denartment of 
Communication, has a contract with the U.S. Department of 
Health, IMucation, and Welfare to help plan formative eval- 
uation of the Federation of f^ocky ^fountain States' Fduca- 
tional Technology Demonstration. Part of our task is to 
compile an historical record of the FTD, in order to provide 
as r.uch useful information as possible for future development 
deci s ions . 

Uliile we currently have a listing of significant 
tlecisions and documents pertnininp to the Demonstration, wo 
would like to insure aj'ainst onittinp any important oroicct 
information from the record. 
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Accord inpl V, we wotild annreciate vour cooperation in 
making the record of significant events as connlete as 
possiMe. A mptno detailing what decision points, f»ile«ttones , 
and docunents seem to vou to be significant to the project 
history would he most helpful, 

('onics of documents could he sent to us in full, or 
simnlv described by general contents, date» to and from, etc. 

Thank you very much for your cooperation," 



Very truly vours, 



N'ancy II. Marklc, Ph.n. 
Research Associate 



.\HM:civ 



I*or their response to this letter, wc thank; 



Mr. Michael fl. Annison 
T>r. Louii? Pransford 
Dr. John Cameron 
Mr. Fred f-brahimi 
Dr. Larry Hrayson 
Pr. Albert Ilorley 
nr. Richard Marsten 
Mr. Arthur Melmed 
Mr. Al Whalen 



Dr. David BerkiPon 

Ms. Pam v'^oui^hlin 

nr. Robert Filep 

Dr. Fdith Crotberp 

nr. Kenneth Lokey 

Dr. Alice Scates 



BEST COPY AVAIWBLE 



Chapter III 
NARRATIVE CHRONOLOGICAL HISTORY 
Conrad G. Carlberg and David G. Markle 



This section of the Chapter gives a chronological 

account of events, which records how the concent in 

Chapter 1 was implemented and upon which the analyses and 
recommendations in Chapter VI are based. 

The history of the Educational Technology Ocmcr strat ion 
is recorded in three periods: an early preparatory period 
from 1968 through 1971 „ a planning period from July 1971 
through July 1972, and a developrAental period from July 1972 
through early fall 1973. 



Early Work: 1968 - 1971 



O verview 

Beginning in 1968 FRMS undertook extensive communication 
with various federal agencies and other organizations, in an 
effort to lay the groundwork for a satellite communications 
nroject in the Rocky Mountain region. Several other agencies 
in the Rocky ^*ountain region had expressed interest in parti- 
cipating in such a project, so discussions focused on what 
was to be done and tlie nature of cooperative arrangements. 

During this period, federal agencies that were interested 
in the development of satellite communications joined in recom- 
mending that frequencies be allocated for use in educational 
broadcas ting. 

Hncouraged by this frequency allocation and by the evidence 
of regional interest in carrying out a satellite comm.unicat ions 
project, on awarded PRMvS a contract for initial planning of the 
demonstration. These activities are detailed below. 



Initial Planning 

In 1968 and 1969 members of F-RMS explored the possibility 
of conducting a satellite-based education project in the Rockv 
Mountain region. Preliminary discussions were held with the 
Office of Hducation (np,) and Communication Satellite Corporation 
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(COMMSAT) (72/O.VlOA, 72/10/04A)4 FRMS staff met with the 
National Aeronautics and Space Administration (NASA), with pri- 
vate companies working on the development of satellites, and with 
education officials in the region to explore their interest in 
such a project (72/10/04A, 71/01/14). As a result of these 
discussions^ FRMS submitted the first of a series of proposals 
to on in 1969. This initial plan focused on imnroving 
educational opportunities in isolated small schools in the Rocky 
Mountain region by means of educational television satellite 
broadcasts. While the proposal was not funded, it led \IVA< to 
contact FRMS two years later concerning a project which was to 
become the PTD. At about the same time the Western States Small 
School Project submitted a similar proposal to OE. The two 
organizations later agreed to cooperate on the project. 

As a result of increased national and international 
interest in social experiments with the Applied Technology 
Satellite-F (ATS-F) (72/10/04A, 71/U1/14) NASA agreed to add a 
2.5 GHz broadcast capability to the satellite. This would 
allow it to be used with low-cost receivers (72/11/16). In 
September 1970, the Federal Communications Commission (FCC), 
agreed to allocate the 2.5 GHz frequency for educational 
television by satellite (72/11/16, 70/09/21) on an experimental 
basis. This policy position was forwarded to the World Admini- 
strative Radio Conference (WARC) via the Department of State. 
This request for frequency allocation represented joint 
agreement among the FCC, NASA, the Office of Telecommunications 
Policy (OTP) of DHEW, and the Office of Telecommunications 
Policy of the White House. The H.S. proposal was accepted 
at the WARC conference held in Geneva in June of 1971 
(70/10/22, 72/11/16) and the way was cleared for the experi- 
mental use of the 2.5 frequency on the ATS-F. 

Simultaneously HFW began to explore potential areas for 
experimental use of the broadcast time that would be available 
on the ATS-F. A number of potential experimenters were 
contacted, including FRMS (72/01/14, 72/11/16). As a result 
FRMS presented a preliminary plan in March 1971. The plan 
included career education in the public schools, early childhood 
education, and higher education. A cooperative effort was plasjied 
among the Education Commission of the States (PCS), FRMS, and 
the Western Interstate Commission for Higher Education (KICHE) all 
of which were developing projects in these areas (7:/10/04-\, 
71/03/24), As a result of interest expressed by these and other 
potential experimenters, DHEW submitted a proposal in Anril 1971 
to NASA for the inclusion of educational exnerinents on the aTS-F. 
By agreement among the chief Administrator of NASA, the 



Source documents are referenced as explained in Chanter T, 
page 1. 
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Secretary of the Department of Health, I'llucation, and Welfare, 
and the President of the Corporation for Public Broadcasting, 
CPB and HEW would jointly determine programming to be used 
on the ATS-F. The experiments would include health and child 
development in Alaska, and occupational skills, secondary 
school science, child development, and inter-university 
networking in the Rocky Mountain region. The child develop- 
ment programming would be based primarily on existing programs 
such as Sesame Street; occupational skills were to be directed 
primarily to American Indian groups; higher educat<on would 
consist mainly of the distribution of existing programs 
(71/04/08). This plan was to be modified numerous times over 
the next two years. 



Development of the First Planning Proposal 

In May 1971, the Office of F.ducation awarded FRMS a 
contract in the amount of $35,678 to "develop and articulate 
the organizational structure and planning to prepare for a 
1973 HFW-CPB-NASA satellite experiment for the Rocky >?ountain 
region." This contract was to run through September 31, 1971 
(71/04/27, 72/02/24). 

Upon receiving the contract, FRMS, WICHE, and FCS held 
regional and national meetings rjf experts in early childhood 
education, public school education, and higher education to 
collect information for the planning report. Regional input 
was obtained from state governments, teacher groups, students, 
business, broadcasters and minority group representatives 
(71/04/30, 72/03/lOA). 

The first major planning doi^ument for the satellite 
experiment was completed and submitted to OF in July 1371. 
It stated: 

1) The Rocky Mountain Corporation for Public Broadcasting 
(RMCPB) would coordinate the broadcast and engineering elements 
of the experiment. 

2) ECS would provide programming directed toward children 
with supplementary segments for parents; the latter would 
include some caretaker training. 

3) FRMS would provide programs in occupational awareness, 
communications skills, and environmental studies. 

4) WICHF would provide academic courses, public service 
education, occupational training, counseling, and communications. 
The communications materials were to include both literacy and 
English as a second language. 
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S) $26 million was requested to carry out the proposed 
activities (71/07/19. 71/06/16). 

The proposal presented a plan to provide educational services 
to the people of the Rocky Mountain Rej»ion which would be 
implemented under the direction of FP.MS. Production of needed 
software would be subcontracted in the region. Tt was assumed 
that 1) existing software would he inadequate for the purnoses 
proposed, and that 2) there would be extensive applications of 
two-way, interactive programming using video, voice, and 
computer interaction. Staff would be hired in each of the 
states in addition to a central staff in Denver. The emphasis 
would be on a "user based service delivery system." 



.Vegot iationa and Planning Period: July 1971 • July 1972 



Overview 

The planning period of the P.TD was characterized bv progress 
toward general goals, by unanticipated delays, and by reductions 
in scope. 

A planning proposal, identifying areas for information 
collection, was developed by the HTD staff and accepted by 
PHEW. The FTD followed up on this planning Proposal by inves- 
tigating literature describing prior experience with prelects 
similar to the FTD, and by collecting data on regional demo- 
graphic characteristics, on regional resources, and on existing 
programming. Contacts were initiated in the eight-state region 
in order to develop preliminary plans for site participation and 
site hardware. Content components were planning general imple- 
mentation procedures. 

Delays originated in a variety of quarters. Policy and 
personnel changes at the federal level required changes in TTD 
plans. Attempts to generate funding commitments for the plan- 
ning and operational phases of the project met with unexpected 
difficulties. At the local level, delays that effectively 
demonstrated the functional interdependence of the components 
of the r.TD forced a re-evaluation of the project's organiza- 
tional structure. Compounding these difficulties wove several 
reductions in the scope of the project. While these reductions 
made it more likoly that the project would meet its Roals, they 
nevertheless required changes in plans and reallocation of 
limited resources. 



Initial UeUuctions in the .Scope of the Project 



miKW indicated that it could not afford the ?26 million 
effort proposed in the Tuly 1971 submission. PHr.W's st^incc 
was that the project should he nn experimental demonstration 
rather than the large-scale service delivery system envisioned 
by the Federation. DHF.W outlined a plan that would focus on 
comparisons of the effectiveness of various technolo^iy mixes to 
reach program objectives C72/10/04A, 72/03/lOA). In August and 
September of 1971, TRMS restructured the original proposal based 
on discussions with tfSOH, HEW, CPB, and NASA. This revised plan 
was discussed with regional education, government, and television 
officials in order to enlist their supnort (71/10/05^). 



In September 1971, there was a planning meeting of the 
FRMS staff, the (iovernor of Idaho (chairman of the '^RMS 
Telecommunications Council), and representatives of numerous 
federal agencies. Following this meeting, the Office of the 
USOr Deputy Commissioner for Pevelopment was to organize funding 
for the FTP. The federal agencies that had attended the September 
meeting were surveyed as possible sources of funds. \o agencies 
outside HFW expressed willingness to provide financial support 
for the proposal, even though FRM^5 staff had originally believed 
that a number of federal agencies would do so. 

In October of 1971, the Commissioner of Fducation for 
nnnw net with representatives from FRMS. F.CS, and WICIIF. At that 
tine he made a tentative commitment of 55 million for H' 1973 
for a more limited experiment. This commitment was made 
while restructuring of the July 19 proposal continued under 
a contract supplement (71/10/26, 72/02/24, 7 /08/12). The 
program content of the project was reduced to earjy childhood 
and career development and defined as an educational technology 
♦experiment' or 'demonstration' (72/03/lOA, 72/10/04A, "^1/10/04, 
"1/10/07, 71/10/26) . This seemingly unimportant use of words was 
to become a major source of confusion, conflict and funding dif- 
ficulties: Was the project an experiment or a ('emonstrat ion? 
IsTiat should its goals be? Who should be rcsnonsiblc at the 
federal level? 

In '.'ovembor 1971, the first of numerous staff chanros at 
the federal level occurred: the Project Officer was chanced 
and the I>i rector of Telecommunications Policv, nilFW, '^Tice of 
Telecommunications Policy was named Project Coordinator. 
("J/OVlOA, 72/\0/'SO) Another contract supplommt was awarded 
to FRMS to continue planning activities (^'l/ll/lS, 72/in/04\, 

'i/io/o-^, •'i/io/:6). 
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Discussions about the project occurred at a time when IlliW 
was going through several policy debates: •'services integration" 
was proposed by the Secretary, nilHW, with the purpose of 
combining health, education, and welfare programs, and "educa- 
tional renewal" was proposed as a method to package discre- 
tionary monies in 01: to support programs for children in areas of 
high need across the country. The creation of the National 
Institute of Education (NIE) , a new agency to he the research 
and development arm of the federal government, was also pro- 
posed. Legislative hearings were being conducted on these and 
other proposals in 1971 and the result was numerous changes in 
policy, operating procedures, personnel and appropriations 
levels- -all of which were to affect the project. 

In January 1972, the revised planning proposal was submitted 
and a planning grant was awarded; $500,000 of the requested 
$800,000 was granted by USOE. According to this planning 
proposal : 

1. The two program areas i^ould be career development and 
early childhood development. Higher education programs were 
dropped, 

2. A production and engineering group would be responsible 
for the satellite and ground communications systems and for all 
production. There would be approximately 300 sites In the 
region. 

3. A utilisation group would "localize programming" for 
participants at the sites and handle public information. Parti- 
cipants would represent all ethnic and geographic groups in the 
region. 

4. Research a. id evaluation would be conducted by each of 
the project groups or components. 

5. Most programming would be new. Some existing materials 
would be used. 

6. Programs would make extensive use of two-way communica- 
t ions . 

The two major participating organizations would be FRMS 
and KCS with support from WICHE and RMCPB. PRMS was the grantee 
responsible for career development, broadcast and engineering., 
and utilization. F.CS was a prime subcontractor responsible 
for early childhood. The proposal constituted an aproonent to 
complete seventeen specified tasks during the planning period 
("2/01/10, 72/10/30) 
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PRMS instituted a management monitoring system based on 
weekly reports of estimated proportions of tasks accomplished 
and amounts of money expended. The system was organized around 
the 17 tasks identified in the January 1972 planning proposal, with 
the responsibilities for the completion of these tasks divided 
among the four ETD components. The directors of the components 
were the decision-making body for the project (72/03/17). Hach of 
the directors was responsibile for activities in his area--career 
development; early .childhood; broadcast, engineering and pro- 
duction; and utilization. Responsibilities were organized 
on these conceptual areas instead of along functional/operational 
lines. This arrangement was to frustrate and produce delays in 
management decisions focused on functions for the following year. 
Conceptual/organizational boundaries did not always reflect 
functional boundaries, even though no component could operate 
effectively in a vacuum. For example, inter-component design 
groups had been organized in August 1972 to develop courseware 
and handle user implementation problems. These were abandoned 
within two months as they required more inter- component coopera- 
tion and information sharing than was obtainable. Another 
instance of this problem was a continuing debate over whether 
Utilization or Content components should be responsible for 
field operations. At one point there were three separate 
field operations - -one in Utilization, and one in each of the 
two Content components. 



Initial Component Activities 

After approval of the plan, the Broadcast and I'nfi inoering 
component met with NAvSA and Fairchild, which built the ATS-F, to 
begin planning the design of ground system equipment. Thev 
also conducted a preliminary survey of existing regional trans- 
mission facilities. A major data source for this survey was 
FTD staff experience in the region. The survey covered PBS 
stations, network links between them, translator systems, and 
cable television systems (72/06/15A). The survey indicated 
that the design and operation of the regional terrestrial 
communication equipment was consonant with the capabilities 
of the ATS-F (72/07/28^) . 

In March 1972, the broadcasting and engineering plan? were 
based on the assumption that the satellite would broadcast to 
public broadcasting stations in the Rocky Mountain region, some 
CATV systems, some translator systems, «nd a few individual 
sites unreachable by existing systems. Planning addressed 
both the quantity and quality of one-way video and two-wav 
video, one channel audio, four channel audio, CAI/CMI, and 
remote uplink video-and-audio mixes to be utilised at tho ^^itcs. 
Snecific tasks undertaken by the Broadcast and Fngincerine 
component included tho identification of satellite ranah i 1 i t i es , 
Planning for down-converter construction, and the identification 



of field sources for broadcasting and receiving signals. Four 
types of site hardware were being considered: basic '^'V 
receivers, an auxiliary mode allowing the reception of narrow 
band signals in lieu of the TV signal, a low cost narrow band 
transmitter, and a single prototype of a TV transmission 
terminal to permit return video from remote locations. 

The responsibilities of the Utilization and Research 
component, in March, 1972 included: coordination of site 
selection, identification and development of contract pro- 
cedures for participating sites, demographic data collection, 
nreparation of research procedures and designs, and the pro- 
duction of an explanatory and public relations brochure. 
'Specific tasks assumed by the component included the collection 
of demographic data, the selection of test sites, the develop- 
ment of a research and evaluation program, the development of 
training models for on-site personnel, and the development of 
further contact*, within each pr.rticlpating state (72/03/17). 

The ntiliz/ition component information collection effort 
began in January 1972 with a three month review of literature 
on utilizption approaches and procedures (72/07/28A), This was 
supplemented in February by a series of meetings with various 
agencies and government units throughout the nation to study 
utilization models and approaches. In February a subcontract 
was awarded by FRMS to WICHF to collect information on 
utilization models (72/02/25). 

The Utilization component also used consultants to collect 
information. In March 1972 one consultant made recommendations 
on a public relations strategy for the FTD. In May, 1972 
another reported on the advantages of using multi - lingual 
field staff in the utilization effort (72/03/lOA, 72/05/11). 

In March 1972 a contract was let through the Offlre of 
the Secretary, nUHW, to the department of Communication, *^tanford 
University to assist the Federation in planning for the evalua- 
tion of the operational phase of the FTD. This effort involved 
assisting in the development of specific objectives for 
the project components, the drafting of instruments to document 
project accomplishments, planning formative evaluation proce- 
dures, assessing tho feasibility of a summative evaluation of 
the project, and providing a general history and analysis of 
the project planning phase. F^etween January and ^^arch 197: the 
content components (Career Pevelopment and Farly Childhood) 
conducted reviews of the literature, programs, and materials 
for possible use in the FTD (72/05/17). 

To furrf\or ri'^sist in planninji, two advisory (»rotms 
wore created. The Satellite Advisory Committee consisted of 
rov'ional husincs'i executives, regional broadcasters and repre- 
sentatives from sub -contract ing organizations. It was intended 
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to provide advisory input to the project as a whole. This 
group met twice. ICS formed the Varly Childhood Technical 
Advisory Group, which consisted mainly of the F.CS Early Child- 
hood task force plus additional advisors from the Rocky ^'ountain 
region, to help guide the component's planning nhase. As a result 
of some disagreement amonp members after the group's first meeting, 
they decided that no further meetings would be held; rather, the 
component would consult with individual members as needed 
(72/02/18). 

FRMS also initiated cooperative work with the staff of 
the Alaskan Project, Areas of planned cooperation were 
engineering and early childhood with some discussions about 
career development (72/04/28B, 72/05/15A, 72/02/09). In March 
1972, after reviewing other projects and studies of field support, 
Utilization and Farly Childhood began the development of a general 
prototyping and field testing plan for validating programming* A 
start was also made on identifying factors to bu included in the 
formulation of site participation agreements (72/06/05). TRMS 
staff asked state Governors to designate a single contact nerson 
in each state. 5 Fxcept in Idaho, where the contact remained with 
the Governor's office, the Governor* designated the Chief *;tate 
School Officer. All subsequent contact in each state was to go 
through those designated (72/10/05). A plan for the organization 
and the function of the state field staff was then developed in 
conjunction with the states. The field staff as originally 
planned would consist of a State Coordinator, Circuit Riders, 
and site personnel (72/03/22, 72/03/27). 

The Utilization staff of the FTD traveled to the 8 partici- 
pating state-j to meet with each Chief State School Officer. 
The purpose of the meetings was to brief the school officers on 
FTD plans and to establish contact between content components of 
the V.TD and the state agencies which were to be involved 
(7;>/04/28A, 72/05/lSA, 72/03/21). These meetings were Part of 
a ••lew profile" public information stance maintained to avoid 
raising false hopes before plans were made final (72/05/15A). 
The continuing debato on the goals, objectives, operating 
plan, and funding level for the project made it impossible to 
make conmitments to states or cities in the region. Tt also 
connounded internal budget decision difficulties. 

Lists of child care facilities in the regron were conniled 
for use in site selection (72/07/28A). As VCC and TRAC deadlines 
approached, and as funding uncertainties continued this list was 
almost continuously altered and resubmi t^-cd , 



IT The six states in the Federation are: Colorado, I'rah, Vew 
Mexico, Wyoming, Montana, and Idaho, N'evada and Arizona also 
were to participate in the Demonstration. 
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The finrlv Childhood component continued to studv existing 
nrogratn resources. This was accomplished bv visits to 
educational laboratories, attendance at teachers' conventions, 
review o** commercial publications and advert isoments, and 
meetings with state and local school personnel. 

In April 1972 the titllization component decided that field 
structures would vary among the states, depending upon local 
environment and conditions. The general basic structure would 
be the same in all the states with key field personnel hired 
by the KTD. These were to include the state coordinator, 
circuit riders, and the site coordinators (72/07/28A). Contact 
was initiated with the Navajo nation through tho Navajo Tribal 
Chairman (72/04/2SB, 72/OS/15A) . 

While the content, engineering, and utilization planning 
was continuing the HTD learned that there would be a delay in 
the satellite launch date of from 6 to 12 months. This delav 
afforded the V.TTi extra time during the developmental phase 
(7:/05/15A). 

The preliminary design of the ground support equipment 
system was completed in April, 1972. Equipment specifications 
and installation costs were determined (72/06/15A). 

Owing partly to an inability to come to agreement with 
funding agencies and partly to a delay in obtaining passage of 
the 1972 HFW appropriations bill, funding constraints had 
plagued the nroject. Project Staff decided that a soecific nlan 
for pre-testing software could not be made, they felt that not 
only were there insufficient funds, but also specific objectives 
and audiences had not yet been identified (72/06/15A). 

The liarly Childhood component was still developing plans 
for innlementat ion . A subcontracted survey of instrumentation 
and measurement in child development was completed and n report 
submitted. Th? purpose was to identify measurement instruments 
that could be used in HTD evaluation (72/05/OU) . In *'av 1 972 , 
subcontracted reviews of CAI/CMI programs existing in child 
development and career education were also submitted f^'J/OS/lSn, 
72/05/150). The purpose of these reviews was to examine existing 
computer based programs that might be suitable for use or 
adaptation by the TTD. 

At about this time, a tentative general implementation plan 
was formulat<»d by the V.arly Childhood Component. This imple- 
mentation plan consisted of caretaker and parent training nro- 
f;rams, a computer-based bibliography of child care techniciues 
and information, and proiections of the , project's intended lonR 
term effect on state certification and coordination procedures. 
This tentative plan was set down in a formal planning document 
by the end of May 1972 (72/05/21). 
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In ^^^y, the managenont monitorinj^ systom had run its 
course for the components which had completed their' pi anned 
tanks and activities. No replacement system was immediately 
implemented. 

During May and .June, 1972, a survey was conducted to 
identify possible signal receiving and uplink sites. This 
survey consisted mainly of examination of data concerninf! 
frequency allocations and the area coverage of these frequencies 
(72/07/28A). 

The staff decided that the production unit should have 
full time availability of basic equipment in order to access 
the satellite (72/07/28A) . Under the original nlan most 
production was to be sub-contracted, and the TTD was to have 
a limited production facility. This procedure was expected 
to distribute both experience and economic benefits thus 
helping to develop regional production capability (72/07/28A, 
72/10/30) . 

After discussions between or and TUMS it was decided that 
FRMS would obtain a minimum capacity in-house production 
facility, rather than implement the original plan. The 
construction of a limited capability production studio was 
not, however, officially approved until February 1973 (72/07/28A, 
72/10/30). The facility would probably consist of electronic 
production equipment, and possibly would include a mobile unit 
for production and transmission from remote areas (72/07/28A, 
72/10/13) . 



Reorganizations, Changes^ and Delays 

In June 1972, after numerous internal discussions and talks 
with federal officials, it was decided to reorganize the resnon- 
slbllities for public Information, research, and internal evalua- 
tion. There would be a director for each of these functions 
located in the Administration component, and staff for each 
function would be located in each of the other components 
(72/10/05, 72/07/28A). 

An attempt was made to align operating and function.il 
responsibilities more accurately. Since each component was 
staffed to j'erform many funct ions -- some overlapping with other 
components but none equipped to handle any overall function 
for the complete demonstrat Ion- -a high premium was nlnced on 
cooperation. The management conflicts that this organizational 
structure produced were not resolved for more than a vcar. 

The staff determined In .lune 19"'2 that a large amount of 
new programming would have to he made for the f-TP project, as 
there were not enough satisfactory programs available in ti." 
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chosen subject matter areas, furthermore, new orogrammini! w«5 
expected to he better because programs would still have to he 
relevant in June of 1974, at the end of the satellite's scheduled 
broadcast time. Proj^ramining was of central importance in the 
continuing discussions of project focus. A general statement of 
objectives for the Career Development component was formulated 
(72/07/28A) . 

Various problems and delays began to surface in Tune. 
Because the taraet audiences had not been specified, and because 
specific operational objectives had not been formulated, it 
was not possible to develop a sampling design or to identifv 
experimental and evaluation variables in detail. One effect of 
this delay was to prevent final site selection. Because sites 
had not been selected the staff felt that they could not complete 
the planning for validating data collection instruments and for 
allocating ground equipment systems. Hach of these Issues was 
compounded by conflicting federal guidance and by the difficulty 
of reaching solid internal project decisions. In order to plan 
the satellite system and a backup system, it was necessary to 
do an extensive survey of terrestrial broadcasting capabilities 
existing in the region. The staff thought that there was not 
enough money to perform the survey. 

On June 1, 1972, the first draft version of the PTD nlan- 
ning report was sent to Washington. On June 16, nHFW requested 
revisions in the planning report, including further reductions 
in the scope of the project and more precise specification of 
project objectives. On June 25, this revised planning renort 
and proposal were sent to Washington. 

In addition, in June 1972 a $300,000 planning grant was 
awarded by the Department of ^fanpower Development and Training 
to explore possible joint benefits for the Federation and the 
nMDT (72/0S/09. 72/06/15n). The Olympus Research Corporation of 
Salt Lake City was a sub -contractor to the Federation, carrvinc 
out part of the work for the DMDT planning grant (72/09/05)'. 

The Department of Labor awarded a $50,000 supnort grant 
to the Federation to explore the feasibility of the Demonstration* 
preparing the way for expanded delivery of POL services on satel- 
lites to follow (72/06/15n). 

The Tune 25th version of the Keport and Proposal was then 
extensively revised and rewritten in accordance with criticisms 
and suggested revisions sent from DUFW on July 7, The second 
revised version of the Report and Proposal was submitted on 
July 28 to OKRW (72/07/28A, 72/07/07rO . The nnF.W Budget had not 
^ccn approved hy the President; no contract was in place for 1973; 
and the project was supported under continuing resolutions and 
an extension of the planning grant. 



I'ntler thi» Plan submitted on .luly 28, the litiUr.ntinn 
component took on three additional responsibilities: 1) it 
would obtain user input and involvement in programming, 
2) it would incorporate HTD programs into permanent structures 
within the Rocky Mountain region, and 5) it would coordinate 
r.TD resources with existing resources in the region (72/07/28A). 

Plans which delineated the steps necessary to obtain proto- 
type sites by October 1973 were made by the tftilization component. 
These included the necessary contact and agreement steps 
(72/07/06A). 

The design groups (mentioned earHfr) composed of staff from 
l.arly Childhood, Career Development, Production, Broadcast and 
lingineering, and the Stanford evaluation planning group, were 
created to assist the two content components in planning broadcast 
content. These groups worked for two months. An attempt was made 
to snecify steps for the empirical development of component 
programs (72/07/28A). 

An ATS-F program review meeting was held in Washington during 
July with representatives from PHEW and CPB attending. The 
technical features of the satellite were presented and described 
and representatives from FRMS, Stanford, and the Alaska and 
Appalachia experiments presented reports on the status of their 
planning to NASA (72/07/OSA, 72/07/06H) . 



Developmental Period: July 1972 - October 1973 



Overview 

The activities which were planned for the PTD developmental 
phase were, with few exceptions, impeded by .i number of reversals. 
These, plus new directives from the federal level, maintained 
the TTD in an almost constant state of flux. 

Funding delays were beginning to affect the project in July. 
Vew staff were brought in for the developmental r)liasc of the 
project. The scope of the project was again narrowed. Clearance 
for an uplink frequency was refused. The lack of specific program 
content caused delays. Directives restructuring a major portion 
of the project were received during a transition of resnons ibi 1 itv 
for the project from OF. to the newly created National Institute of 
Fducation. A group of observers from N'lF reviewed the progress 
of this project in April during a site visit in nenvcr; their 
assessments were predominantly negative. The project was ex- 
tensively restructured. A full time Project Director was 
appo inted. 



Funding Delays 



By the end of July, 1972 anticipated funding delays iTaposed 
restraints on planned HTI) activities. The VW staff decided to 
concentrate on the design of instructional material, the develop- 
ment of field ^import staff, and the implenentat ion of a new 
PT-RT management information system until funding availability 
allowed necessary travel to the states and sites (72/08/07B ,0; 
72/07/31). 

Beginning .Tuly 1st the V.TD was supported on a month-to-month 
basis under a continuing resolution. In August 1972, the President 
vetoed a bill authorizing the new HEW budget. The veto meant that 
even if an agreement could be reached with the funding agency, 
the r.TD would continue to be funded under continuing resolution 
and therefore at a limited level at least, until Congress re- 
passed the appropriation in amended form (72/07/28A, 72/07/28C, 
72/07/31. 72/10/30, 72/11/20). 

In August, the component Directors submitted estimates of 
staffing needs for the developmental and implementation phases. 
Necessary administrative support services were identified and iob 
descriptions were prepared. In the continuing discussions on 
management, a decision was made to form a separate Production 
component, removing Production from the Broadcast and Engineering 
component. The new component's primary responsibility would be 
to coordinate and monitor subcontracted production (72/08/08A, 
72/10/13). 

The I '^cat ■ on of the regional uplink and origination and 
delay center had not been determined. Two options existed: 
1) the Federation might own and operate the center; 2) one of 
the PBS stations in the region might handle these functions. An 
agency to handle overall liardware systems and integration for 
satellite educational experiments had not 'oet-n designated. This 
function could be subcontracted out or it could be handled either 
by the ETD, the Appalachia or Alaska experiments, or the health 
component (72/06/15A). A decision to have RMS do it was made in 
August 1972 (72/10/11). 

NASA decided to give the Broadcast and Engineering romncnent 
of the ETD the responsibility for the operation of the master 
station; in effect, this step made ER^^S responsible for the 
satellite educational communications system (72/10/11). The 
unlink and ori?!ination center for the ETD would be operated by 
the Broadcast and Engineering component, as it was decided that 
tho PBS stations in the region could not afford to perform 
this function (72/06/15A. 72/10/11 , 72/08/25R). 

In September 1972, DHEW requested that the ETDplace less 
emphasis on content and more emphasis on the comparison among 
various delivery systems and technological mixes (72/09/1 4'\) . 
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It also requested a more brief and specific statement of the 
operational plan and scope of the P-TD (72/10/02). 

After numerous discussions with federal officials the 
Addendum to the report and proposal of July 28, 1972 was completed 
and sent to Washington in October- -well into the fiscal rear. 
The purpose of the Addendum was to distill from the larger dorn- 
ment a succinct and specific statement of F.TD plans for use in 
obtaining funding for the development period (72/10/02). 

A very general research design for the project was formulated 
as part of this document. Data analysis would employ the site 
as the experimental unit. About 60 sites would be studied in 
depth: 30 "intensive" sites and 30 "non- inteu^ ive" sites 
(72/10/02). This plan was never fully agreed upon among the 
f'.TD components. 

Representatives from NASA visited Denver and briefed the 
entire F.TD staff on deadline requirements with reference to the 
data of satellite launch (72/09/14A) . 

During September, as funding became available, meetings were 
held in the states with representatives of the Chief State School 
Officers. The major purposes of these meetings were to interview 
the applicants who had been selected by the states for the 
State Coordinator positions and to prepare hiring agreements. 
Site selection was also discussed and reports on the progress of 
the V.TD were presented (72/09/25, 72/09/28, 72/10/05). 

In late October 1972, State Coordinator training meetings 
were held; because of time constraints the training period was 
shortened to one week instead of the planned six weeks ("2/10/02). 

State Coordinators made site visits in their respective 
states in order to formulate recommendations for final l-TD sites 
in November. Ten 2-way sites would have to be selected by 
November 15 to meet PCC clearance for filing deadlines (72/10/26R, 
:'2/10/26F, 72/11/ 22). 

It was necessary to file with the FCC by November IS to 
obtain permission to operate the microwave link needed by the 
FTD. In order to file, however, detailed technical information 
was needed. There was no money available to collect the needed 
data (72/10/11). 

The President signed a continuing resolution authorization 
for HFW. This action meant thnt the FTD would have to be funded 
bv I{nw under continuing resolution for the developmental year 
(FY 1973) ("2/10/30). The impact of this was to con- 
found the program and contractual relationships with niillW, the 
Office of Fducation, and potentially with the newly formed 
National Institute for Fducation. The financial problems. 
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• combined with the consistent inability to reach agreement on 

the goals and scale of the project continually frustrated 
decision processes within the project and at the federal and 
state level. 



Additional Problems 

Toward the end of 1972 both the Early Childhood and Career 
Development components were conducting limited needs assessment 
surveys. The purpose of the surveys was to identify needs common 
to the region as a whole and needs unique to certain areas. This 
user input was intended for use in making specific programming 
decisions (72/10/26A, 72/10/26F,) . Instead of being one of the 
first steps taken in project planning, this activity wa? going 
<;n eleven months after the first contract had been awarded. 

Finding a director for the Production component had been very 
difficult. The search had been continuing for the four months 
since the August decision and the delay in filling this position 
produced undesirable delays in production decisions and activity 
(72/10/1.3, 72/10/30). In late November 1972 , a director for the 
• Production component was hired, as was a Research Coordinator 

(72/11/22). These two appointments completed the senior staff. 

« A list of more than 300 potential sites was sent to v.'ashington 

for clearance through IRAC and FCC in December (72/12/04A) . 

In August of 1972, a new and more comprehensive management 
information plan was implemented. A full-time staff member 
had been hired to oversee the system; 1500 to 2500 activities 
were to be monitored by the system. Most of the completion dates 
for PERT tasks and activities were set back one month or more 
during December. The major cause of the scheduling delay was 
the inability of the content components to specify program con- 
tent. This factor delayed Broadcasting and Engineering activities, 
site selection, production planning, evaluation planning, and 
program scheduling (72/12/04A) . In late December of 1972 it was 
becoming clear that unless content was specifically defined and 
complete scripts drafted V' / soon it would be impossible to meet 
either the prototyping deadline of March 1973, or the satellite 
deadline of April 1974 (72/12/26). 

Early in 1973, the primary audience for the Career Development 
component wa^ narrowed from grades K-12 to grades 7-9. Career 
personnel and project monitors had agreed that the audierce should 
he narrowed because of time and resource constraints. Crades 
7-9 were chosen because many states in the region already had 
career development programs in grades K-6. A secondary audience 
was to be adults in the region who are responsible for adolescents 
in home and institutional settings. 
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In January it became evident that, for reasons of national 
security, TRAC might not clear the frequency that was to be used 
for the uplink from intensive terminals (73/01/26A). This uplink 
was to be the return link for two-way broadcasting. 

The Harly Childhood component had decided what the user 
needs were and what objectives and competencies were to be 
taught; production on the first of the Career Development modules 
had begun (73/01/26A). 

Prototype sites had been selected by the Htilizatinn component 
and visits to the 274 tentatively selected demonstration sites 
were being planned in order that information for final selection 
decisions might be collected (73/01/26A) . 

In February the OK staff notified FRMS that a number of 
changes in the project would be necessary. These included a 
limitation of video courseware to live or short lead time produc- 
tion; the inclusion of as much interactive capability as possible 
in the courseware; the restructuring of content component staffs; 
the confirmation of specific remote site audiences; and the selec- 
tion of a full-time project director. (The President of FRMS had 
been directing the project on a part-time basis, with the assis- 
tance of an Assistant Project Director). FRMvS was also instructed 
to proceed with the creation of a median capability studio 
facility and to purchase equipment to access the satellite and to 
equip a network control center (73/P2/27). 

Immediately following these directives (March, 1973) was 
the news that IRAC had refused to grant the clearance for 
broadcast to the satellite from remote sites (73/03/21A). 

The February directives from the Office of Hducation led 
to the cessation of all new script production for the Fnrly 
Childhood component and for the Career Development component 
(73/03/21A), These events were inconsistent with usual guide- 
lines for both federal and grantee activity. Fvery aspect of 
the project was affected by these events and negotiations about 
goals, objectives, and implementation proceeded in a far less 
amiable environment. 

Ry April 1973, the satellite communication design had been 
completed by the Broadcast and Fngineoring component. The Career 
Develooment component was structuring and outlining their script 
production; no scripts had yet been written. The Farly Childhood 
component was completing scripts in a modular format. The Pro- 
duction component was releasing studio equipment bids and identi- 
fying available production talent for eventual staffing of the 
studio. The Utilization component was finalizing its prototyping 
Plan (73/04/25). 

The Farly Childhood component was prototyping its input/ 
feedback model by April, 1973. The Production component had 
selected a location for ita studio, and the Utilization component 
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had visited 701 of the nominiited receive-only sites, Script 
production in the Cnreer Pevelopment component began. The 
Harly Childhood component had forwarded fifty minutes of scripts 
to the Production component. Production activities were, however, 
in "hold" status pending funding decisions. 

The activities during April and May, 1973, are described 
above as they were listed in the minutes of the management's 
Status/Heview meetings for the two months. How much actual 
progress occurred during this time is not clear. The predominant 
atmosphere among much of the T.TJ) staff at the time was one of 
uncertainty. Few personnel were certain for how long, and if, 
they would continue to hold their jobs. A visiting team was 
assembled in April by and NIE for a site review of rro^ress. 
The overall response of the panel's members to the nrojcct was 
critical of most components and components' progress (73/04/15, 
•3/04/16, 73/04/17, 73/04/18A, 73/04/19A, 73/04/19B, 73/04/190, 
73/04/20A, 73/04/20B, 73/04/23). One of the results of the 
panel members' visit was a strong insistence by the National 
Institute of Education that there be a full-time Project Director. 
The Assistant Project Director was ineligible for the position, 
the Director of the Broadcast and Engineering component was 
named to be the Project Director. 

During late May and early June, the new Director began nego- 
tiations with a new federal monitoring team for Fiscal Year 1974. 
As a result of these contract negotiations, the ECS Earlv 
Childhood sub-contract was eliminated and the Career ^evelop- 
nent content area was retained. The Career Development conponent, 
one of the areas that was reduced, was placed within the Production 
component. Further reorganization established a new Research 
group. Budget cuts as well as staff resignations had reduced 
the number of staff from 100 to 64. 

At the beginning of Fiscal Year 1974, the project was 
operating at a substantially reduced level. Discussions were 
continuing among officials and t^rms on the nature and scope of 
the project. By October 8, 1973, when the Stanford Hniversity 
Contract ended, funding had still not been settled for FY 1973- 
1974. Another issue that was still unresolved was the potential 
use of two-way communication. 
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Chapter TV 

HISTORY PROM THH HTD COMPONFAT FUNCTION VIF.WPOINT 

Conrad C. Carlberg 



In this Chapter a discussion is undertaken of the 
functions and KTD activity of (a) the Content Comoonents » 
(b) the Utilization Component, (c) the Broadcast and 
Fngineerlng Component, and (d] the Management and Monitoring 
System. 

This chapter represents the viewpoints of PRMS staff *:o 
a greater degree than do other chapters in this report. The 
author worked, to a large extent, from documents prepared 
by FRMS staff rather than from direct observation. Many of 
those documents are informal and undated. 



Content Comnonents 



• Both content components collected information for program 

planning during the planning period. Mthough much of the 
FTD's information collection activities were regionally based, 
some did not involve the nocky Mountain region. The waior 
sources of this information were written data archives and 
consultants . 

In February, 1972, the Farly Childhood component exneri- 
mented with the establishment of a Technical Af^visory Croun 
to provide information on program design and evaluation in 
the early childhood area. This group met foi the first time 
on February 18. As a result of a disagreement with F.CS 
staff, p,.rticipants decided not to meet again as a group but 
to make themselves available on an individual basis for 
consultation as needed. Members of the group had been drawn 
primarily from an FCS task force committee on early childhood. 

Beginning in March, 1972, and continuing for the next 
three inonths, existing program resources in the area of early 
childhood education were investigated. The early childhood 
component staff visited regional educational laboratories and 
teachers' conventions, and they reviewed commercial catalogues 
and advertisements, niscussions were held with state and 
local school personnel. The program resources identified 
were examined for their potential use in the FTD (72/Oii/21). 

A portion of the review of existing program resources 
was subcontracted. The report of this subcontractor, which 
was submitted in early May, reviewed CAI and CMI pro^^rans 
existing in the early childhood area. The purpose of the 
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review to examine existinr, conputcr-bised progranis that 
rir.ht ho suirf.Me for use or n'lapt.-ition (72/05/153). 

Another report on subcontracted information collection 
was also subnltteJ in May. This report identified measurement 
instruments that nif»ht be used to evaluate early childhood 
efforts (72/n5/01A). 

The career development component's information collection 
activity proceeded on a pattern similar to that of the early 

childhood comnonent . During January and February, n review 
of research literature in the area of career education was 
conducted. In this same period the major portion of a review 
of existing career development programs and Instructional 
materials was conducted. For this review, consultants were 
brought to Denver and career development component staff 
made site visits to career education centers around the 
nation. In May, a report was received on a subcontracted 
survey of existing CAT and CMI programs on career education, 
fn April, a survey of regional public and private resources 
in career education was initiated. Throughout the entire 
planning period, a survey of existing career education 
measurement instruments was conducted by the component- 
staff (72/05/15A, 72/07/28). 

Sever?.l naior restrictions to the scope of the TTD were 
made in May, 1972 , just prior to the preparation of a planni -.g 
report and proposal in June and August, At a meeting with 
representatives from state government agencies concerned 
with early childhood, a final decision was made to restrict 
the scope of programming in the early childhood area exclu- 
sively to adults with child care responsibilities. \o pro- 
grams aired directly at children as an audience would be 
prepared. The decision was based on state preference and on 
information that had been collected during the planning 
period. 

At the same meeting, plans to implement a computer based 
bibliograph'/ of child care techniques and Information were 
mentioned, and ways in which the PTD might effect long term 
improvements in state early childhood care certification and 
coordiiiat ion procedures were discussed (72/05/21). 

Two major restrictions in the potential scope of the 
career developm.ent portion of the project were made tUiring 
the planninj; period. Hnrly in tl\e plannmj^ period the possi- 
bility of ii'.cluding some actual training in specific voca- 
tional skiU'^ in the p roj^ramni nj; had been considered. Later 
i t v; a 0 c i d o d i ii s t o a '.I to r t a i n t h e e x elusive program 
enpha-^is on career information and counseling functions, 
becau.ie actual vocational training was beyond the capabilities 
of the r.TP. 



o 

ERIC 



4..; 



$9 



A nnjor restriction uas nndc in the scone of the 
niuUcncr the carct»r ilovclopment nortion of the nrojcct would 
address. The project had intended to provide career educa- 
tion oT jjrndcs K-12. Hovrever, after ^HMS/on discussions, a 
decision was made to restrict the audience to adolescent 
students (73/01/19A). 

Seven major tasks were identified by the f-arly Childhood 
connonent as preliminary to actual content nrogrammin?: a 
preliminary review of the literature, the listing of alterna- 
tive behavioral objectives, the selection of behavioral objec- 
tives, a continuing literature review, a continuing analvsis 
of an early childhood development data base, a review of 
existing software and hardware, and an analysis o^ instrumen- 
tation needs (72/0.^/17). 

Tentative career education research variables had been 
identified and listed by March, 1972. The independent 
variables included population considerations, delivery 
systems, and supplementary assistance as categories. The 
dependent variable categories included attitudes and skills. 
A review of career development literature and media was 
instituted. Identification of notential production facilities 
was undertaken and a rough outline for prototype nrojiramming 
was completed (72/03/17 ). 

During Aoril and May of 1972 , contact v.'as made throughout 
the region with various individuals and agencies that might 
provide useful input to the i-TD, and with representatives 
of the eventual user population. Early Childhood component 
staff members met with television station dpoartm.ents of 
education, state Departments of Education, educational broad- 
casting companies, and data processing companies. User 
nooulation representatives included, among others, the 
Navajo Tribal Council, the Nevada State School Superintendents, 
and the Arizona State P.T.A. (72 /05 /OlB). 

On Vay 20, 1972, the Early Childhood Development Com- 
ponent held a conference in Reno, Nevada. Connonent staff 
met with the Early Childhood Development designees of the 
Chief State School Officers to address regional and state 
issues in early childhood development. Issues which were 
reviewed and critiqued included considerations pertinent to 
government and institutions, teacher training, parent involve- 
ment, licensing and certification, consumer attitudes, and 
children's programming (72/05/21). 

During August of 1972, two design groups, one for each 
connonent, were named. Desit^n grouns were intended to be 
nultidiscinlinary and thus included representatives from 
Engineering and ntilization (72 /08 /01). 
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The lii iiv Childhood design proim settled on a dovclon- 
ncntal process noi]c\ consistint! of four phnses: develon»^ental 
testing, nrototyping, pilot testing, and demonstration. Tho 
tvnes oF baseline information and the evaluation tochniaues 
necessary for the development testing phase were discussed. 
The groiin considered the fa':t that available time for identi- 
fying, hardware, software, ..nd personnel variables was growing 
short. Another issue was inducements for parents to come 
to a central receiving site. If the programming Itself were 
not sufficient inducement, the major Harly Childhood audience 
night have to be reached at home: this would mean program 
cltstribution by cable and I-TV network outlets. The" latter, 
however, would not commit themselves to participation unless 
<''nd until they could pass on program quality. A third issue 
that received continued attention was the question whether 
one could specify behavioral or measurable objectives for 
early childhood education. The design grouo decided that 
the production process should begin in early September, 1972, 
beginning with a set of objectives that had already been 
developed (72/08/33). 

The design group effort had initially appeared promising, 
nroviding as it did a chance for improved intercomponent 
communication. However, the Early Childhood design group 
did not proceed beyond the initial discussions, owing to 
uncertainties about group leadership and authority. The 
career development group met intermittentlv , but eventually 
ceased to exist. 

Field coordinators for each state had been selected by 
October, 1972, when a meeting was held for the purposes of 
state coordinator orientation and training. At this meeting 
it was decided to undertake a needs assessment in early 
November to obtain user input for program design. This 
assessment would consist of interviewing three tynes of 
people in each state: (1) parents, child care teachers and 
staff, and community leaders; (2) professionals (universitv- 
based experts in child development) ; and (3) government 
agency staffs. These interviews were to be conducted by the 
State Coordinators (72/10/26A). 

The needs assessment for the Career '^^evcloonent connonent 
was to bc;un in the middle of N'ovember. The asscssnent was 
to be acconplished by the use of an instrument that had been 
pretested in October. It was to be administered to junior 
high and hij^h school students by the Career nevelonm'»nt staff 
(72/10/26A) . 

The F'^jruary, ]'y-7> directive from the n<^fice of !".'ucation 
hroui^ht abnut major changes in both the structure and function 
of the content components. Courseware was to be designed to 
include as i?.uch interactive capability as possible, using 
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primarily dif»ittil feedback. The emnhftjiiji in production wns 
changed to primarily live and short lead time prograimnin}?. 
Thus, the staffs for the two content areas and associatctd 
production staffs were to be reduced. Script production In 
both content components came to a halt in February. Content 
comnonents turned their attention to revising objectives and 
scripts that had already been nroauced, restructurinp the 
body of program content, and creating test items from content 
objectives (73/03/21A). 

A further revision in content plans came about in early 
1973, when NASA indicated that the time required to realign 
the satellite between the footprints was greater than exnected, 
thus cutting time available for F.TP nrogramminp from twenty 
to fourteen and one-half liours ner week (73/03/21A]. 



Prototyping 

Prototyping considerations had surfaced by .Tulv of 1972. 
Population and accessibility characteristics of orocotype 
sites were being discussed, as were the program prototype 
criteria to be fulfilled at the sites (i.e., the amount of 
HTD program and the number of subjects to be involved). 
The site selection procedure and the training and functions 
of support personnel were under consideration. It was also 
noted that information was needed on the selection criteria 
for prototype sites, the number of subjects needed at che 
sites, a specification of the program content at the proto- 
type sites, and a training schedule for state coordinators 
and circuit riders (72/07/08). 

But by December of 1972 it was clear that time con- 
straints would rule out the use of programs produced in-house 
during the prototyping period scheduled for March of 1973. 
It was also apparent that it would be difficult to obtain 
subcontracted programs for the prototyping period. It was 
suggested that the prototype period be postponed until 
materials to be tested had been prepared (72/12/04B). 
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The responsibilities assumed b>* the Utilization and 
Research comnonent at the outset of the VT\) nroject included 
coordination of site "election, identification and devclon- 
ment of contact nrocedures at each site, preparation of 
research nrocedures and desi>jns, and design of a public 
relations brochure. Tasks included collection of demo- 
granhic data, selection of tejt sites, the development of 
a research and evaluation program, the development of train- 
ing models for on-site personnel, and the development of 
further contacts within each participating state (72/0.'S/l 7) . 

The Utilisation component information collecMcn crfnrt 
began in .lanuary 1972 with a three month review of the litera- 
ture on utilization approaches a.nd procedures. This staf^- 
conducted literature review WhS supplemented in FebrMarv bv a 
series of meetings around the nation with various agencies and 
government units to study their varying utilization models and 
approaches, in February a subcontrsict was awarded to t'no Western 
n.terstate Commission for Higher T.ducatiop to collect information 
on alternative nossible utilization mode's (72/02/25). 

Collection of information about the repion was started 
in February with a survey of regional demographic data. \ 
review of potential resources and constraints on a rcRional 
utilization effort was also initiated in February. 'Regional 
information collection continued into April 1972, when a 
survey of Pns services in the region was conducted (72/10/05). 

The Utilization component also made Ufe of consultants 
in information collection activity. In March 1972 a con- 
sultant's recommendations on a public relations strategy for 
the FTD were obtained. A Mav 9 consultant's report analyzed 
the advantages of nult i - 1 ingual field staff in the utilization 
effort (72/03/lOB, 72/05/11). 

Planning continued through Anril of 1972, when efforts 
were directed to implementing utilization plans, \ nrogran 
for initial training of the state cadre in each of the 
nart icinatine states had been dcveloned. Tt included an 
orientation to the project, role responsibilities, d.itn 
evaluation and collection procedures and models. An overview 
of the content comnonents was nrei^ared, as were a description 
of broadcasting and engineering considerations and identifi- 
cation of media resources and their utilization (''2 /04/24), 
In addition, contacts for Planning utilization nrocedures in 
each of the eicht >'tates werf i-rogress ing. Followun contact 
nectinf's were conducted durinp. the month of Nnril in t^enver 
to discuss snecifically each :;tate's specific needs and 
ideas. These meetings were attended by representatives from 
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the ^en.irtment oT lUlucation and Social Services, designees 
from other selected agencies, and meinhers of the Tarly rhild* 
hooU nevelonmont compon'int staff (72/04/28A). 

Puring Septetnher and Octoher of 1972, the tttllisation 
staff nade visits to New Mexico, tftah, Montana, Tdaho, and 
\ri:ona in order to present an update of TTD progress, to 
exnlain the current thinking on state structure and the 
position State Coordinator, and to discuss criteria 
for site selection. These discussions were held with 
representatives of the Departments of Tducrtinn of each 
state (72/09/25, 72/09/28, 72/10/16). 

Utilization planned three different field sunp» rt 
systems: high, medium, and low levels, Tt also develoned 
nlans for prototyping! field support systems and for the 
provision of field information services. 

Utilization, as is noted above, was responsible for 
the coordination of the site selection process. The process 
was initiated in March of 1972, when a list of all possible 
sites was drawn up by Utilization. These lists were taken 
to the Office of the State Superintendant of Schools in each 
state for revision? and suggestions This process continued 
through April, May, and Uine, when conjponent representatives 
were discussing the nominated sites with state agency staff. 
It was felt that this give-and-take process had pleased the 
states: they felt they had provided real input into the 
site selection process (72/10/05). 

A \ASA deadline briefing on September 14, 1972 required 
selection of thirty intensive sites by November, 1972 (72/09/14A). 
State Coordinators were to identify sites for installation 
of two-way terminals for prototyping operations and for needs 
assessment. Py November 15, nine or ten sites were to have 
been selected ^or FCC clearnnce of two-way equipment. 
Selection was to be based on ethnic and cultural characteris- 
tics, language variables, geographic variables, personnel 
aval Inbi 1 i ty , and installation selection factors. However, 
in memoranda which resulted from a visit by USor representatives 
in December, 19'^2, it was noted that there was considerable 
redundancy in the sit***^ selected: two-thirds of the ■^ite? 
were located in areas already being seiVed bv PHS stations 
(■^2/12/18) Hy March, 1975, this difficulty had apparently 
been corrected and a site agreement plan w.is being (!ra**ted 
(73/03/21A/ . 
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Technical Questions 

The initial technical focus of the F.TD was on use of 
two-wav audio and video for interaction v/ith the audience. 
Computer assisted instruction was also under consideration. 
As a result of funding constraints as woll as non-monet nry 
planning decisions, this focus has been changed to include 
possible use of a computerized data bank and digital trans- 
mission of data, with only limited two-way audio and fes or 
no two-way video applications (73/01/19A) . Subsequent changes 
have further limited the technical possibilities. 

In March, 1972, the Broadcast and Tngineering component 
indicated that its planning was based on the assumption 
that the satellite would be broadcasting to public broad- 
casting stations in the Roc!.y Mountain region, CATV systems, 
translator systems, and to individual sites. Planning 
addressed both the quantity and the quality of one-way 
video/two-way video, one channel audio, four channel audio, 
and CAI/CMI remote uplink video-and-audio technological 
mixes to be utilized at the sites. Specific tasks undertaken 
by the Broadcast and Engineering component included the 
identification of satellite capabilities, planning for down 
converter construction, and the identification of field 
sources for broadcasting and receiving signals. Four hard- 
warf» configurations were also being addressed: basic TV 
receivers, an auxiliary mode allowing the reception of narrow 
band signals in lieu of the TV signal, a low cost narrow 
band transmitter, and a single prototype of a TV transmission 
terminal to permit return video from remutc locations 
(72/03/17). 

Bv late October of 1972, the V.TD was planning on 2nn 
receive-only terminals (one-way video) at a cost of $3,000 
each, 31 intensive sites (two-way video) at a cost of $4,000 
each, and 76 two-way audio sites at a cost of $6,500 each. 
These included 46 sites to be used in the Alaskan experiment 
and 15 sites to be used in the Appalachian experiment, for 
which the P.TD Broadcasting and Pngineering component had 
been given the responsibility of integration and implementa- 
tion (72/10/25B). 

Beginning in December of 1972, a number of pronlems 
began to surface. Postponements became obligatory, ^ne 
scheduling delay involved the interdependence between nlans 
nadc bv the Broadcast and Fnp i neering- component and thf» 
production of program content by the f-.arly Childhood and 
Career Pevelopment components. On the one hand. Broadcasting 
and Fngineering had a number of fixed deadlines for cquinmeni 
purchase and installation which had to be met and which were. 



in part at least, dependent on content component plans. On 
the other hand, the content component needs assessment efforts, 
and their desire not to specify content before the needs 
assessment was completed, had delayed their content specifi- 
cations. Further, the bids on the receive-only terminals 
ran much higher than had been expected, and difficulties in 
receiving clearance from International Radio Allocations 
Commission (IRAC) for the intensive terminals had caused 
delay in their construction (72/12/04A). 

Another major source of schedule slippage was the necessity 
for TTD coordination with the Alaska and Annalachia experi- 
ments. Coordination was necessary before a final satellite 
schedule could be determined and NASA was requesting this 
as soon as possible. However, the Alaska and Appalachia 
projects were even further behind in specification of 
program content than was the FTP (72/12/04A) 



Frequency Allocation an<' flearancc 

In 1971, the Department of Health, Pducation, and 
Welfare, the National Aeronautics and Snace Administration, 
and the Federal Communications Commission cooperated in a 
successful effort to obtain from the World Administrative 
Radio Conference an allocation of 2.5 GHz for broadcasting 
of educational services via satellite (71/04/08, 71/n4/14B). 

One cf the unique aspects of the ETD was to be the 
provision of interactive, two-way communication between the 
point of program origin and the remote receiving sites. It 
was assumed by project planners that the 2.25 CHz would be 
available for broadcast from the two-way (intensive) -^ites 
to the satellite, thus providing the return link to the 
point of program origination (73/0.3/21A). 

This assumption was mistaken. In January, 1973, 
the International Radio Allocations Commission, one of the 
agencies from which clearance to use the frequency for unlink 
purposes was necessary, indicated that it might refuse to 
grant frequency clearance for the proposed intensive sites. 
This naturally caused deadline problems for the FTD. N'ASA 
required that intensive terminals be installed by October, 
1973, for simulation tests. However, it made no sense to 
issue Requests for Proposals for construction of the intensive 
sites until clearance had been received from IRAC. The clearance 
w<?s officially refused in March of 1973 (73/03/21A, 73/01/26\) . 

This left NASA nnd the FTDwith several unnalatahlc 
alternatives, nne was to redesign the VTS-r satellite so 
that it could handle another frequency for two-way inter- 
action. This alternative was dropped when such redesign 



was found to be too costly in money, time, and manpower. 
Another alternative was to use the two existing satellites, 
ATS-1 and ATS- 3, for interactive communication. This was 
not a narticularly happy alternative, because both satellites 
were then operating beyond their life expectancy and failure 
could have occurred at any time. In addition, the sound 
quality delivered by either left much to be desired and 
video transmission was not possible. The compromise of using 
ground lines for short distance transmission and using 
existing satellites for long distance transmission was 
considered (73/03/21A). As of October 1973, the questions 
regarding two-way transmission were not resolved. 



Management Monitoring System 



Efforts to design an information monitoring and 
accounting system for the project began in January 1972. 
This system was to identify all tasks, means of accomplish- 
ment, staff responsibility, costs, and time requirements. 
The system was based on seventeen basic project tasks which 
had been identified in the January 10, 1972 planning pro- 
posal. It was linked with an automated accounting system. 
Information vas fed into thi» system by means of v/eekly 
component reports of monies expended and percentage of tasks 
completed. These progress fitjures were compared v:ith 
Planned progress figures. A work-completed and cost-to-date 
ratio would then define the progress made on each task in 
weekly reports. The system was intended to enabl<^ project 
administration to monitor all planning activities, make 
decisions at appropriate times, and to modify planning 
approaches as necessary. A summary of tasks, progress, and 
costs to date appears in the March 17, 1972 Progress Report 
(72/03/17) . 

In May this information system was terminated. It was 
noted that the tasks and anticipated work flow defined 
in the management information system formed a theoretically 
sound base for the operation of the project. It was also 
noted, however, that day-to-day operational needs were 
occasionally frustrated by this theoretical base, which 
tended to build inflexibility into the on-going operation 
of the project (72/03/17). Additionally, task activity was being 
concluded in preparation for the writing of the June Planning 
Report and Proposal. A further reason for termination was 
that the ETD staff had expressed disappointment in the 
system's ability to reflect actual project activity accurately, 
its flexibility, and its capability to facilitate internal 
communication and task coordination. These problems were 
attributed to the fact that the system had not been staffed 
on a full-time basis and to the fact that it had not been 
automated. The need for a useful system was, however, still 
apparent (72/01/19B). 

Accordingly, in Auc:ust of 1972, implementation of a new 
and more comprehensive management information Plan was 
started. The second version of the management information 
monitoring system was a more ambitious undertaking than the 
first. A full-timo staff member was hired to oversee the 
system. In addition, the system was automated. Snec i ^icat ion 
of the 1500 to 2^00 activities to be monitored by the new 
information system began in August. The system's output 
would be 1) a PHRT-fhart, consisting of a computer- pranhic 
showing the order to tasks and their inter- relat ionships , 
and 2) reports on performance and progress accomplished which 
would result in adjustments and resource allocation and the 
restructuring of activities (7:/08/08A). 



In a federal memorandum dated August 23, 1972, comment ing 
on the FRMS JuTy 28, 1973 Progress Report and Proposal, 
it was noted that the latter document lacked a work plan and 
budget for the next project phase, and that a work- flow PF.RT 
chart did not constitute such a work plan C^2/08/23). 

By October 28, 1972, a bi-weekly opportunity to change 
tasks (if necessary) was made available. A coding system, 
in the final stages of preparation, would cross-relate PERT 
tasks, sub- tasks, and phases with cost items in order to 
determine how much each activity cost. The accounting system 
was to be based upon daily time sheets filled out by each 
KTD staff member. In addition to reporting their own time, 
State Coordinators were to report consultant time and nart- 
time help obtained from state agencies (72/10/30). 

By October 30, 1972, the design of the PERT system was 
complete. Implementation was to begin in mid-November, 
when the accounting codes, which were the reporting basis 
of the system, were completed. The extent to which the 
PF.RT system was successfully implemented and proved to he 
of functional value was seen as potentially important for 
other projects. PERT techniques had not previously been 
extensively applied to a large scale social project (72/10/30), 

By this time the number of basic tasks monitored by 
the PERT system had been reduced to 14. The system utilized 
the IBM Project Management System (PMS IV) (72/ll/17r.) . 

In early December 1972, the PERT system had made obvious 
the need to change various estimated completion dates of 
tasks and activities. In the rescheduling process, the 
effect of PERT was to clarify interdependencies in task coia- 
pletion schedules. Failure by one component to complete a 
task as scheduled required the "slippage" of completion 
dates of tasks in other components. The major pressure to 
adhere to a firm schedule came from the Broadcast and 
Engineering component, whose tasks were inherently most 
specifically defined and whose deadlines were most unavoidably 
final (72/12/04A). 
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ANALYSIS OF THE HISTORY OF THF FEDERATION OF ROCKY 
MOUNTAIN STATES' EDUCATIONAL TECHNOLOGY DEMONSTRATION 
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Introduction 



The FRMS ETD was conceived by on-site planners as a 
project of large scale and scope that would solve many 
problems for many persons in the Rocky Mountain region. At 
the same time it was seen as an information gathering effort, 
rather than a problem-solving effort, by some federal Planners. 

Trying to put together a project that is large m 
scale and scope as well as highly innovative is a difficult 
policy and operational undertaking. Experimental investi- 
gations and evaluations of costs and effects are difficult 
to carry out in such settings where maintaining standard 
procedures and holding variables constant is operationally 
difficult and may be see. as inconsistent with larger social 
goals . 

Early in its planning some federal observers felt that 
the ETD was imbalanced in the direction of too large a scone 
and too much emphasis upon service delivery. It was honed 
that the externally funded on-site evaluation planning 
team would help the FRMS staff to reduce the scone of the 
project effort to an evaluable size; this was explicitly 
stated in the Stanford contract but was found not to represent 
a consensus among all parties involved in planning the Demon- 
stration. The evaluation planning team was not sufficiently 
influential to effect much change in project plans. The 
desired reduction in scope was achieved only at great cost 
later, after ^he visit of a site team from the National 
Institute of Education in mid-April, 1973. 



so 



While decisions must renain open about the relative 
merits of significantly large social exnerimc-nts , which arc 
difficult to evaluate, and small controlled experiments, 
which often lack major impact, a significant amount of 
information has been gained from this nrolect. 



Rationale 

Kverv Project has unique experiences and problems. 
If the experiences and problems of the FTD were all unique, 
the writing of history, analvsis, and recommendations would 
be a pointless task. Fortunately, information has been 
compiled over the past ten years on a variety of projects 
whose problems and experiences either are mutual or are 
gencralizable across related settings. 

Some of the early experiences in nlanning for an 
F.ducational Technology Demonstration in the Rocky Mountain 
region can b#» examined in the light of other projects. We 
have chosen to compare the ETD with three other innovative 
instructional media projects: one in PI Salvador (Schramm et 

• al. 1970; Mayo 5 Mayo, 1971), one in Colombia (Comstock and 

Maccoby, 1966; Comstock et al , 1966), and one in American 
Samoa (Nelson, 1970). These three projects and the 

, FRMS Project have planning, developmental, and operational 

implications for both policy making and direct implementation 
of future similar projects. 

The theme of the first groun of recommendations is 
the expectations between project and funding agency and 
within* agency and project, other recommendations deal with 
continuing relationships as the nroject grows and matures. 
Intra-agency perspectives are examined in the context of 
the way these may affect the nroject. Still other recom- 
mendations treat operational considerations. 



Insuring That Kxpectations Match: 
The Grantee and the Funding Agency 



Recommendation 1 : Before significant funds are expended, the 
Grantee an< the funding agency must agree on the goals and 
obiectives of the project, the procedures to be used, level ol- 
funding, developmental and imnlementat ional procedures, and 
nethods for arbitrating differences of opinion that nav 
develop during the project. 
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necotnmenaAtion 2 ; To the extent nossible, the nv-mt or contract 
Jocunents should reflect the initial expectations of all 
individual and organizational narties involved in the rrant 
ne.eot iations . 

Recommendation 5 : In order to maintain sunport, nroiect staff 
must be able to demonst* that ideas are clear, that the under 
taking is financially n -ole in a fixed time frame, and that 
the results will contribute to the solution of the problem for 
which the project was conceived. 



The FRMS Project. 
Background 

The President of the Federation of Rocky Mountain States 
was the first project director for the Educational Technology 
Demonstration, fie assumed the position with the understanding 
that he would serve in that caoacity until he could identify 
a full time executive. By January of 1972, operational 
responsibilities were delegated to four comnonent directors: 
one for Broadcasting and Engineering, one for Utilization, 
one for Career nevelonment, and one for Early Childhood. 
The Parly Childhood com'^onent was a subcontract operated bv 
the Education Commission of the States but residing on-site 
as an integral part of the Project. The other components 
were comprised of Federation employees. The four Component 
directors had nominally equal respiansibility and authority. 
More complete background information may be found in Chanters 
I -IV in this Report. 

Early Expectations 

A major problem with the early phases of the FRMS ETD 
was that the expectations of the funding agency, NASA, and those 
of the FRMS project initiators did not match. Coals and 
funding were subjects of continuing disagreement (discussed in 
detail later in this paper). 

Policv makers in the funding agency and in other 
cooperating, agencies originally conceived the FRMS KTP to he 
a somewhat limited demonstration of the way satellite tech- 
nology can be applied to educational problems. Tt was also 
considered important that some allocation of frequencv on 
the ATS-F satellite be made for social nurnoses in addition 
to the technical, scientific purposes that satellites were 
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• alreadv fulfilling (71/06/02, 02A, n2B) . Pederal nolicv 
makers aia not envision, as did FRMS staff, develonment of 
numerous hours of courseware, or the establishment of 
extensive br jadcasting, engineering, or nroduction facilities 
in the Rocky Mountain region. 

The Education Commission of the States subcontract 
comiionent for Earlv rhlldhood Development programming had a 
director and a number of staff members by February 1972, 
According to (72/02/24), from the Acting Director of the 
National Center for Educational Technology, to the Denutv 
Commissioner for 'lenewal, ECS was given the resnonsibil itv 
for initial planning of the Demonstration. Staff from 
VRHS Components also wished to influence plans. Tn reality, 
Planning progressed as a large-scale, and somewhat frag- 
mented activity. The Early Childhood Component, as a result 
of its charge, exercised considerable influence on the plans 
and on the manner in which the planning funds were expended 
during the second planning grant. At this early time, a 
pattern of component-centered rather than project-centered 
activity developed. This pattern seriously damaged the 
project in the funding agencies' view. There were senarate 

• attempts, for example, to include field services, res'^arch, 
and evaluation in the activities of the Early Childhood, the 
Career Development, and the Utilization Components. These 

• separate component activities were not coordinated across the 
project. Although the redundancy of these efforts was due 

in part to poor coordination and communication, it anpears 
that it reflected also the desire of individual components 
to dominate as many functions as possible, thus increasing 
the importance of that component. A feature of this activity 
was that many staff were hired too early in the planning phase 
for the Project. 



Mismatches Concerning Funds and Scope 

During the planning and early operations neriods, the 
funding agency made many requests for the FTD staff to 
rewrite proposals and to shift project emphases (71/10/07, 
72/07/07, 72/12/15B, 73/01/08, 73/02/27, 73/04/19D, 
73/06/OlB). These requests were based upon a number of 
factors: (1) FRMS proposals were judged by reviewers nor 
to be adequate planning documents. (2) There was an unresolved 
mismatch in expectations between project personnel and 
the funding agency. (3) Some agency personnel felt that 
the project was spending or would spend more than could bo 
justified by the expected return of either information or 
delivery of services And, (4) some agency personnel ^elt 
that the project staff could not spend the requested funds 
competently in the available time. Additionally, as a result 
of federal reorganizations and of budget impoundments by 
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the White House, there were immediate pressures and constraints 
on money. Pocument (72/12/15B) from the Federal TTD Project 
Co-Monitor to FRMS Managements, says, "...there Is obviously 
not <1 2, 000, 000 available to cover such materials development... 
a decision will probably have to be made to reduce this 
fij?ure..." From (72/12/18), "The projected budget for FY 
1974 should be re-estimated. The figure of $6.5 million 
that you requested does not appear reasonable." 

Further strengthening the funding agencies' view that 
FRMS staff were attempting to accomplish too much within a 
short period of time, Document (73/04/190) says, "...the 
staffing and planning of the production component are inade- 
quate to the task of producing 200 hours of programming in 
the required time frame of 12 months." 

Production was only one of the continuing noints of 
contention, although it is the most outstanding substantive 
issue. Others concerned the adequacy of planning in its 
entirety. 



Goals and Objectives 

The FRMS proiect staff had considerable difficulty in 
clearly stating their goals, objectives, and procedures. The 
following excerpts from document (72/07/05B) from a NTFT 
staff member, transmitted through the Acting Associate 
Commissioner, NCET, to the Associate Commissioner for Fduca- 
tional Technology exemplify the result of this difficulty: 

"...After reading some 300 pages of material, t ^ind 
myself unable to comprehend, with any clarity, the 
objectives to be met; the programs which will be created 
to' meet them. . .Too often, the mere assertion that some- 
thing will be so, is offered as a substitute for a 
delineation of how the asserted result is to be achieved. 
The report does not address itself to the expected 
impact, effects, or benefits of these programs, in terms 
of the defined needs." "There is no indication of how 
the sizeable investments in materials collection and 
production will be used after the satellite experiment 
has ended." 

Federal reactions of this tyne demonstrate the extent 
of problems that can result from failure to agree at the 
outset on what is to be done and how. 



Patterns 

As a result of the lack of an original consensus on 
Project goals and objectives, an unfortunate "came pattern' 
became established early: The Project personnel would 
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write a proposal for costly, large scale experimentation 
and delivery of social services; in response the federal 
personnel would ask them to justify the costs, explain 
scheduling, provide plans for programming, or make a new nro- 
posal. The Project personnel would then respond either with 
an elaborate justification for their requests or with a new 
costly proposal. The federal personnel responded to these 
efforts by again asking for justification of costs, explana* 
tion of schedules, plans for programming, and so forth. 

As was discussed earlier, Project efforts suffered 
because separate Con«Monents attempted to define independently 
the universe of utilization, content development, experimenta- 
tion, and service delivery. These attempts made the Project 
budgets unjustifiably large. Furthermore, they were not 
coordinated with one another, so that different sections of 
FRMS proposals said different things about what looked to 
reviewers like the same activities. 

We conclude that the multiple director operational 
procedure (four Components, each with its own Director) did not 
work as well as might be hoped. There are at least three 
possible explanations for this lack of success: First, it may 
have been simply the personalities and competencies of the 
individuals cnosen for the director positions. Second, the 
directors were not well acquainted at the start and had not 
worked together previously. Third, the organizational 
structure encouraged each director to establish an indepen- 
dent sphere of influence. 



Proposals 

For whatever reasons, (and some may be derix'ed from the 
above discussion), the Project did not prepare pood reports 
and proposals. They suffered from several flaws: 1) Too 
large and too costly an effort was proposed for the length 
of time available in which to prepare for it. 2) As a 
result of the lack of coordination between Components, budgets 
were large. To at least some of the federal personnel the 
budgets even seemed "padded." 3) Although each proposal 
was very long and gave activities of Project personnel in 
detail, critical elements about operations, procedures, 
schedules, and plans were left unstated. This last practice 
was often justified in personal communications to the Stan- 
ford personnel from the Project staff ^ the notion that if 
concrete plans were stated the Projecxwould then be held 
to them unreasonably by the funding agency. In fact, it 
seemed to many observers that the Project staff either did 
not have clear plans or did not know how to write a reasonable 
proposal . 
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Document (71/10/07) from t > Associate Commissioner, Bureau 
of Libraries and Educational Technology to the rommissioner of 
Education, says: 

"The key fact that ATS-F wi)' be able to serve the 
Rockies for less than a year and is only an experimental 
satellite has been largely overlooked in the proposal. 
Instead of testing, experiments and demonstrations, the 
proposal is devoted to a sizeable software production 
effort. Consequently, one of the key reasons for ATS-F, is 
that it will build knowledge which could in 10 years lead 
to a multi -channel satellite for educational usage, has 
been largely neglected. 

"Secondly, the proposal does not make adequate use of the 
special capabilities of satellites. Nearly all of the 
programs suggested could just as well be shown via the 
existing public television and cable systems. Projects 
in which the satellite is cost effective, such as reaching 
the large isolated and migrant populations of the area, 
need far more emphasis. 

"Moreover, the proposal does not explicitly consider 
some of the interesting technical experiments, such as 
interactive learning, that should be tested using a 
satellite." (A more detailed critique of the oroposal 
is appended) . 

Document (72/07/07B) nearly f* year later than that quoted 
above, address'^d to the President of FRMS from the Acting 
Associate Commissioner, NCET, says: 

"I . General Concerns 

"Purpose, goalS| and objectives . The overall purpose 
of the project is unclear in terms of goals and 
objectives. Review results indicated uncertainty 
as to not only project goals and objectives, but to 
specific component objectives and tasks. 

" Time schedule . Because of the time constraints imposed 
by the launching of the spacecraft, and because of 
the massive undertaking the Federation has proposed, 
a time schedule or PFRT chart should be carefully 
developed for each component and for the total project 
management. The schedule should be realistic in 
terms of necessary training, production, and proto- 
type testing, but should allow the flexibility needed 
for component areas to develop quality programs. 



" Report format . It was 'iXtremely difficult to follow 
eacn section in an overview due to the variance in 
component reporting. An Outline for the entire 
report should be developed that would serve as a 
guide for each component section. 

" Decision-making . Although decisions which were made 
during the planning process were presented, no clear 
discussion of either the sources of information upon 
which the decisions were made, or the reasons behind 
why and how the decisions were arrived at is 
apparent in the report. This concern is particularly 
crucial if critical decisions have been made concerning 
the various constituencies to be served and those 
constituencies have not been consulted during the 
decision-making process. 

"In line with the above, supportive data for the 
decisions made would also give credibility to the 
needs and thus to the programs proposed to meet those 
needs . " 

nocument (72/08/07T)) from a Field Reader of the TRMS 28 July 
1972 Report and Proposal to the FRMS Project Officer says: 

"...There are no specific programs of instruction 
described and most of the space in the report is taken 
up with long boring descriptions of the bureautic 
structure of the project (i.e.. Utilization fomponent, 
Early Childhood Compbnent, Career Education Component, 
etc.). 

Document (72/08/22) from the Acting Director, Division 
of Technology Development (DTD) to the Acting Associate Commis- 
sioner, NCET, concerning the same FRMS report, says: 

"As with their previous submission, this document 
contains a great deal of verbiage and little content. I 
am certain that they could have presented their ideas- - 
and done so more effect ively- - in one tenth of the space. 
After reading the document, it still is not cleai to me 
why they are doing what they are doing. They have not 
yet built the case for the need for early childhood and 
for career education — other than assuming that they are 
needed- -nor have they shown how their approaches will 
alleviate existing problems in the Rocky Mountain region. 

"In addition, although this is an educational and not a 
technical experiment, I do not see the need for a 
satellite. In short, I can not determine from their 
document what they wish to experiment with or what they 
hope to achi'»ve as a result of an experiment." 
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Document (72/08/23B) from the Acting neputy Director, HTD, 
to the Acting Associate Commissioner, NCHT, says: 

"The proposal suffers from the lack of a statement of 
who is to do what, delivered how, to whom, for whac 
reasons, to reach what results, under which conditions, 
for how much. Without such information it is difficult 
to understand on what basis one would fund a continuation 
effort. It might be more practical to fund the oppor- 
tunity for the contractor to state in explicit terms 
just what is to be done and with what level of commit- 
ment from whom." 

Document (72/08/25C) from a staff member in the office 
of Program Planning and Evaluation to the Acting Associate 
Commissioner, and a Staff Member, NCET, says: 

"...Although this document represents a substantial 
improvement over the previous version, it does not, in 
my opinion, provide an adequate basis for further funding. 

".. Apparently there are no specific goals and objectives 
identified for the Demonstration. These are absolutely 
necessary so that all of us, including tax payers, can 
understand why the Federal government is investing more 
than $20 million in this project. The overall goals are 
also necessary in order to impose some discipline on the 
remainder of the project. At present there is little or 
no relationship among the objectives specified for Parly 
Childhood, Career Development, Utilization, etc. If 
some specific goals and objectives had been set for the 
entire Demonstration, there would at least be greater 
coherence among the parts." 

In commenting about the problem with coordination of 
efforts. Document (72/07/07) to the President of FRMS from the 
Acting Associate Commissioner of NCET, says: 

"Although the document [FRMS' 25 June 1972 Report 
and Proposal] presents utilization as the coordinating 
component cf the project, each component section (of 
the proposal] speaks to such tasks as training, field 
services, research and evaluation. Not only would this 
appear to dilute the effort of the utilization 
component, but more importantly appears to necessitate 
excessive staff support if each area would require trainers, 
researchers, evaluators, etc. ...it is unclear why the 
early childhood development and career development sections 
address evaluation apart from the Stanford responsibility. 
This would clearly seem a duplication of effort and 
expense. ...Seldom do .the individual sections identify 
how they relate to each other, i.e., utilization to early 
childhood, broadcast to career education. ...Information 
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riathering and Hisscminat ion should he a separate comnonent 
or part of ♦ftiliiatlon, tvhy one for rvc and rcn? Ditto 
for ricld Services. Research and Hevelopmcnt should he 
a project wide effort and not Imhedded a^; «iiih-sets of 
each cluster, ^ftich common data collection, etc. I" 



Mismatches '^ctwecn the fundinq a^cncv in.' -m r.u . : 
Initiators are likely to occur at the concoptuMl 'tnco of uiv 
project. Many of these mismatches may not heconc an^^nront duriny 
initial negotiations unless all parties make their positions 
clear. It is necessary to clarify positions initially, since 
tacit expectations remain sources of potential confrontation 
at later times in the project life--when there is the risk that 
it cannot survive them. Initial positions are often difficult 
to clarify because Inter and Intra organizational oatterns of 
communication have not been established, and rules for establish- 
ing patterns of communication ar-' 'Ot clear. In addition, some 
individuals or agencies may have .easons for restricting communi- 
cations- -hoping by this means to gain some advantage in future 
• negotiations or operations. Nevertheless, clear communlcaion 

is required. 

Developing and sustaining commtmication networks to insure 
appropriate and adequate information flows for nlanninp purposes 
is one of the most important steps in management. Clock (1961) 
has written about some of the conditions necessary to obtain and 
apply social research. He says, "The Importance of effective 
internal coinmunlcation In this regard cannot be exaggerated." 



Insuring that i-xpectations Match: 
Interagency Rel Ationships 



Recommendation 4 : The grant or contract documents should 
reflect agreement between the policy and operational levels of 
the funding agency, both of which should be involved in the 
ongoing monitoring and operation of the project. 



The FRMS Project 

There were varying pressures on the project as a result 
of policy, administrative, and personnel changes within niirw, 
N'ASA, and fPB. By June of 1971, agreement to pursue a series 
of demonstration experiments to "...test various educational 
and health applications of communication satellites" (V\S\ News 
Kolcnso No: 71-105; (71/06/11), had been made among VASA 
(71/0tv'14A, B) , liliW (71/06/02 . 02A, 02B), and CPB (71/06/021^, 
:i/0(./14B). 
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Howevtr, earlv in 1973, the Secretary of MF.W was moved from 
his post to that of Secretary of t^efcnse, and in August 19?2, 
the President of the Corporation for Public Broadcasting 
resigned. These changes, which had nothing originally to do 
with the FRMS F.TD, had a significant effect upon the later 
fortunes of the project, since they removed to other positions 
two interested parties of a high policy level. 



General Discussion 

An example of problems occuring from lack of communication 
between policy and agency is cited by dlock, who notes that 
several agencies funded social research during World War II, 
but there is little practical effect of this research. He 
explains the failure as partly owing to the fact that "..♦within 
these agencies, an effective relationship between the monitoring 
staff and the policy makers was rarely established." Apparently, 
after the initial requirements were established, the monitor 
usually did not hear from the policy maker until the final 
report was submitted. By then, "The results were seldom closely 
related to the requirements, and implementation could be 
accomplished only infrequently on the t-^sis of the final report 
alone." 

The recommendation to clarify initial expectations is 
important because during the life of any project conflicts can 
develop within or among the policy or operations parts of the 
funding agency. Furthermore, when the funding agency is 
governmental (especially federal), changes in administrations 
can bring about profound changes in policy and agency behavior 
that can complicate situations and relationships that were once 
fairly simple and straightforward. 



Funding and Arrangements 



R ecommendation 5 ; Once goals and objectives are agreed upon be- 
tween agency and project, the original grant or contract agree- 
ment should make certain that projects are sustained long enough 
to insure "...that the program as a whole does not suffer from the 
pressures on all discretionary programs to shift foci to reflect 
the apparent priorities of the moment." (NCERD, T969) 

Recommendation ^ : Government sponsored projects (anti others 
similar) would i'deally be funded on a project basis, based on a 
well articulated, mutually agreed-upon proposal or work statement. 
Major milestone objectives would be set for review at reasonable 
times. Multiyear projects could operate under these criteria, 
with an overall general project review being scheduled to 
take place once each year. 
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The FRMS Project 



History of Funding 

Document 72/02/24 from the Acting Pirector NCF.T to the 
Deputy Commissioner for Renewal details ^RMS funding and 
negotiations up to 24 February 1972: 

"On .Tune 8, 1971, a contract was let to the Federation 
of Rocky Mountain States, Inc. to develop a series of 
planning documents which would describe requirements 
for the use of a satellite in the Rocky Mountain region. 
The contract was awarded for $35,678 under Cooperative 
Research discretionary funds and covered the period 
May 1 to September 30, 1971. As a result of this 
contract, the Federation submitted a proposal for an 
educational satellite demonstration for $26,284,627. 

"A supplemental contract was awarded on October 1 to the 
Federation for $21,808 and covered the period to 
November 30, 1971. This supplemental funding allowerl 
the Federation to continue its Plans for the development 
of an organizational structure and to modify and re- 
structure the proposed demonstration. 

"In October, Commissioner Marland met with rerpesentatives 
from the Federation and its sub-contractors, the Education 
Commission of the States (FCS) and the Western Interstate 
Commission for Hi;:her Education (WICHE) , the Deputy 
Commissioner for Development, and representatives from 
the Bureau of Libraries and Fducational Technology/ 
• Division of Fducational Technology. The purpose of this 
meeting was to discuss the project and its scope of work. 
At this time, a commitment for $5 million, FY' 73 funds, 
was made to the Federation by Commissioner *<arland to 
carry out the demonstration. 

"In December, Arthur D. Little, Inc. prepared a document 
under the direction of the Office of Telecommunications 
Pclicy/HHW, with the cooperation of the Federation, out- 
lining the basic plans for the health and education 
components of the demonstration. Recommendations for time 
allocations for the Rocky Mountain and Appalachian 
regions and the Slate of Alaska were also included. This 
document was submitted to NASA under the signature of 
the Secretary of DHEW and the President of the Corporation 
for Public Broadcasting. 

"On February 1, a contract was signed for $SOO,000 and 
awarded to the Federation, v/ith a sub-contract to ^'f'S, 
to develop the operational plans for the educational 
satellite demonstration. This contract covers the period 
from .lanuary 1 to August 1, 19/2." 



61 



• We note that FRMS Planners consistently requested funding 
that was significantly larger than any of the funding agencies 
planning documents indicated was desirable or even available. 
Because the original arrangements for scope and level of 
funding for the Project were not agreed upon, the ETP record 
contains many documents relating to attempts to establish levels 
of funding and attempts to secure funds that Project staff felt 
had been "promised" but, for example, not anproved by the 
Congress or not released by the agency in charge of the Proiect. 
Document (71/10/05B) from the Assistant Secretary for Planning 
and Evaluation, DHEW, to the Commissioner of Education, says: 

"4. The general budget figure for this experiment from 
HEW and OE as well as such other Federal agencies 
as may desire to participate should be a minimum of 
$6 million to insure that we achieve our experi- 
mental objectives and have a creditable nroiect. 

"...It is proposed that the $5 million for the software 
development come from the Educational RfiD, Libraries and 
Educational Technology budget. As you know, the 
Secretary will be sending to 0MB a request for addi- 
« tional funds for Libraries and Educational Technology... 

At this time, of course, we have no assurance that nMB-- 
and later the Congress- -will anprove this increased 

• funding. If they should not allow these additional 

funds, we would like to have an understanding with you 
that $5 million for the satellite experiment be made 
available from whatever R§D funds are finally approved 
for HE." 

Document (71/10/07) from the Associate Commissioner, Bureau 
of Libraries and Educational Technology to the Commissioner of 
Education reflects the initial disagreement between DflEW (as 
represented by (71/10/05B) above) and FRMS. The Document says, 
in part: 

"...The Federation of Rocky ^^ountain States, in 
conjunction with the Education Commission of the States 
and with the Western Interstate Commission for Higher 
Education (WICHE), has suggested a $26 million effort 
that places prime emphasis on the development of TV pro- 
gramming software to be used in a satellite experiment." 

The memo expressed dissatisfaction with the joint 
proposal (as quoted on page 55 of this paper) . Owinp to this 
dissatisfaction, the federal agency staff proposed an alternative 
effort to the one given in the FRMS proposal. The federally- 
proposed alternative is discussed, continuing with (71/10/07), 
as follows: 




"The outlined experiment will cost about $6 million over 
three years as follows: 

$3 million for in-school and preschool software 

$1.5 million for 500 ground receivers, 10,000 student 
interactive terminals, and other hardware 

$1.5 million for administration, broadcast and 
engineering, utilization, research and evaluation. 

To fund this, it is suggested that USOE fund $1.5 million, 
other Government agencies fund $1.5 million, and the 
Rocky Mountain region match the Federal contribution by 
funding $3 million. The U.S. Office of T-ducation's con- 
tribution would then be $500,000 a year for three years." 

The memo (71/10/07) ends with a formal request to the 

Commissioner: 

"It is requested that you set aside $500,000 per year 
toward to cost of this project to be effective whenever 
agreements between the parties involved have been 
reached." 

Document (72/04/04) from the Secretary, HEW to a member 
of the White House Domestic Council, says: 

"The planning stage of this project began on January 1, 
1972 and is scheduled for completion on August 1 of this 
year. The Office of Education has awarded $500, POO for 
this effort and plana further funding of $6 mittton for 
the developmental and operational phases. "^ 

Document (72/06/23) from the Acting Associate Deputy 
Commissioner for Renewal, Office of Education, to the President 
of FRMS says, 

"This will confirm your conversation with the Commissioner 
as to the support of the Rocky Mountain iMucational 
Technology Demonstration project during the initial 
months of Fiscal Year 1973. 

"Our present ex, ectation is that the Congress will nass 
a continuing resolution for the initial funding of the 
Office of Education's programs in Fiscal Year 1973. 
We intend to support the Rocky Mountain project on a 
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month- to -month basis by continuing the funding of the 
Fiscal Year 1972 planning grant under the continuing 
resolution. This support will begin as of July 1, 
1972. 

"Once the Congress has passed the full Fiscal Year 
1973 appropriation for the Office of Fducation, we hope, 
subject to mutual agree^nent on a satisfactory plan, 
to provide approximately $6 million for thig project 
in n$oal Year I9?3»"i 

Nevertheless, that FRMS proposal still anneared lacking 
to many reviewers. For example, Document (72/08/25C) from a 
staff member in the Office of Program Planning and Fvaluation 
to the Acting Associate Commissioner, and a Staff Member, 
NCET, says: 

"...It is my understanding that this document represents 
the basis for funding the Demonstration':, production 
phase which would last approximately one yepf and cost 
about $5 million, although this is not explicitly 
stated. Although this document represents a substantial 
improvement over the previous version, it does not, in 
my opinion, provide an adequate basis for further 
funding." 

As was stated earlier, there were changes in the 
monitoring of the FRMS Project and in the agency resoonsible 
for it. Document (72/12/18) to the President of FRMS from the 
Director of the Division of Technology Development reflects a 
continuing disagreement about funding for the project: 

"1. The projected budget for FY 1974 should be re- 
estimated. The figure of $6.5 million that vou 
requested does not appear reasonable and does not 
reflect earlier discussions with FRMS. Although the 
Office of Hducation can make no commitment to a firm 
budget figure at this time, for planning purposes, you 
should develop the FY 1974 budget based on the same 
total sums as available in FY 1973. That is, plan on 
^750,000 being available for the Broadcast and Fngineering 
activities, and $3.2 million being available for the 
remainder of your activities, including Farly Childhood 
and Career Development content development. Production, 
Utilization, and other areas of expenditure neoeasary 
far the completion of the project.""^ 



This letter, written 6 months after (72/06/23) and 
8 months after (72/04/04) (both quoted above, page 15) 
reflects considerable deviation In Intention from that 
expressed by original policy planners. Reasons for this have 
already been discussed, hut reviewed here, they include: 
1) FRMS requests for even larger sums of money than originally 
planned by policy makers. 2) Criticism of F^MS Proposals 
and Plans by reviewers. 3) Non-coordination and non-cooneration 
within FRMS activities. 4) Changes in position of high-level 
Federal personnel. 5) Constraints on federal funds, engendered 
by a long delay in obtaining a signed federal budget. 

Mismatches between the FRMS Project and its fcieral 
monitors in planning and attainment of goals, objectives, and 
project milestones continued. Document (73/01/08) to the 
President of FRMS from the Oirector, Division of Technology 
Development, says: 

"We were pleased to learn that your staff is hard at 
work developing answers to the questions that were nosed 
In my letter to you of December 19, (we believe this Is 
(71/12/18), quoted above! and to the points raised [In 
the) addendum of December 20. All future f>V. support Is 
contingent upon satisfactory answers to these questions. 
\Vhlle there Is a payment of $700,000 that is now being 
processed as part of the FY 1973 grant, all other 
payments and requests. Including the recent reauest 
for the production equipment, will be held pending the 
resolution of the questions. We are pleased that you 
will have a draft of your response to us by January 15. 

"As you develop your response, you will be making 
many very basic assumptions about the direction that the 
project will take. These will have significant effects 
on your FY 1973 and FY 1974 budgets. It, therefore, is 
expected that part of your submission will be revised 
budgets for these two years. As you develon the budgets, 
it will be heloful to tie your project expenditures to 
specific products, objectives, and/or ta«»ks. This will 
allow everyone to ascertain your progress in terms of 
your expenditures." 

Goals rf the FRMS staff and the fundinp apcncy continued 
to differ. For example, Document (72/07/07) to the Presir^ent 
of FR.MS from the Acting Associate Commissioner of S'CMT, says: 

"The expense of production would pnhiVtit extensive 
development of programing, much less allow for 
quality products .... 

"The complexity of establishing a ground system for 
the demonstration would appear to prohibit the use of 
staff time and resources for production purposes. For 



this reason* the Federation should give serious 
consideration to the availability of existing 
rfoduction facilities within public broadcast 
stations in the Region. " 

As it became more certain that agency responsibility 
for the project would change from Office of Hducation (in 
early 1973 the OE National Center for Educational Technology) 
to the new National Institute of Hducation, the agency appears 
to have become increasingly apprehensive about the continuing 
disagr<?ements with the FRMS Project on questions of size, 
scope, goals and objectives, and funding. Document (73/02/27) 
from the NCET Grants Officer, the Director of the Division 
of Technology Development, and a staff Program Specialist 
reflects action taken in the federal agency as a result of this 
apprehension* Sections pertinent to funding are quoted here: 

"After considerable discussion among the NCF.T and OE 
staffs, thorough review of the reports of the NCFT 
consultants, and thoughtful evaluation of the FRMS 
reports and of your reauests for an in-house production 
facility, we have decided that a change is required in 
the scope of the Rocky Mountain ATS-F Educational Tech- 
nology Demonstration. This change is a direct reflection 
of what NCET believes needs to be included in an experi- 
ment that is intended primarily to demonstrate the cost 
effective delivery of technology-based, educational course- 
ware to relatively small numbers of television- isolated 
persons, who may only be reached through a satellite. 

"...It follows from such a re-examination of purpose, that 
it is not possible to justify an expenditure of several 
millions of dollars for the creation and production of 
materials intended to serve the numbers of persons esti- 
mated to be reached via the ATS-F satellite in the Rocky 
Mountain Region. Further, even if this magnitude of 
expenditure was approved for the creation of video course- 
ware, with the intent that this material later could 
receive national distribution and exposure, NCET would be 
forced to conclude that FRMS probably could not produce 
the materials with the high quality and in the quantities 
necessary to justify this magnitude of expenditure. This 
conclusion is based on the time constraints, the mix and 
compet*^nce of the FRMS staff, tne facility situations, 
and FRMS management coordination problems. 

"Therefore, this is to inform you that, beginning 
immediately, all on-going activity not consonant with the 
changes spelled out below must cease, and that an intensive 
planning effort which reflects these changes is to commence. 



"5. Funds for the activities under the *'Brottdcast 
and Hngineering" component > since they are provided 
under separate contract with the HEW Office of the 
Secretary, are not affected by this change. Budgets for 
both Utilization and Administration will be re-examined. 
/ total of $2,000,000 for production of courseware , 
including production facilities and staffing, and further 
content development will he the maximum funding allowed 
for all such activitiee during the remainder of the 
project, beginning immediately and extending through the 
end of FY 

"9. Based on the above, you should immediately take 
steps to minimize expenditures not related to this ne 
effort, and should terminate all unrelated and unnecessary 
subcontracts and orders... 

"11. FRMS should prepare a technical proposal with 
budgetary support for this changed effort, to cover the 
remainder of FY 1973 and FY 1974. The subject proposal 
and budget support should be submitted to \CET/op in 
two separate parts, one to cover the remainder of PY 
1973, and the other for FY 1974, In addition, FRMS 
should also submit separately a preliminary plan and 
budget for FY 1975. The subject proposals and budget 
materials are required to be submitted to the Office of 
nducation for review, approval or disapproval, by 
Anril 2, 1973." 

This action was strong. In addition to the disagreement 
with FRJtS, it underlined a policy conflict between the nn 
Agency and DHEW itself, concerning how directive an approach 
to OHFW funded projects the monitoring agency should take. 

Document (73/04/06) from the Grants Officer, the nirector 
of the Division of Technology Development and the Program 
Specialist, to the President of FRMS, reflects the continuing 
discussion: 

"While these documents covered the same points requested 
in our letter, we find after careful review that they 
present more of an alternative plan rather than a 
response to the outlined requirements. We would be 
pleased to review and discuss any alternative suggestions 
you may have, however, it still is necessary for ^RMS 
to respond to all points in the letter of February 27. 



Hur italics. 
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"One of the major points that we wish to call apain 
to your attention is the requirement for PHMS to present 
a plan for the production of courseware and for the 
content development, within a maximum funding limit of 
aporoximately $2 m.Uion for all such activities during 
the remainder of the project, beginning February 27, 1973 
and extending through the end of FY 1975. 

"You have stated on several occasions that you would 
like OR/NCHT to present a plan for those activities within 
the $2 million limit. While we believe that there are 
several ways that this can be done and urge you to develop 
your own plans, we are enclosing one suggested budget. It 
should be emphasized that we are not stating that you 
must accept this approach, but only suggesting this as one 
idea that you can modify, within the $2 million limit, 
to fit your needs and desires. 

"It may be helpful for FRMS to understand the normal 
relationship that exists between the Office of Fducation 
and its grantees. It is the Office of Fducation's respon- 
sibility to set funding levels and to specify the general 
parameters for all projects that it supports. The grantee, 
in turn, is responsible to supply the specific project 
objectives, and the methods and procedures to be used to 
achieve those objectives, subject to the technical review 
and approval by the Office of Education." 



Funding-Related Results of Site Team Visit 

As a result of the recommendations which were based on 
the results of 12 April site visit by a team from NIE, made 
in nocument (7S/04/24) (from the Director, Division of 
Technology Development to the Associate Commissioner for 
Educational Technology) the Early Childhood Component was 
eliminated from the FRMS Project. That document says, in part: 

"S. The Early Childhood effort should be totally 
changed. The present sub-contract with Fducttion 
Commission of the States should bp cancelled and 
all of the T^resent development efforts terninated. 
Instead, the Early Childhood effort should be 
restructured to* provide remote audiences with 
programs such as Sesame Street, Electric Company, 
and BC/TV." 

Document CS/OS/OS) continues the discussion between 
the Director, Division of Technology Development, the I.TD Project 
Co-Officer, and the \ssociate Commissioner for Educational 
Technology. It says: 
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*'As you will note, we estimate the total 2 year cost of 
the project to be slightly less than $2,300,000.*' 

Document (73/06/OlA), from the Director, Division of Technology 
Development, through the Associate Commissioner for Fducational 
Technology, to the Deputy Director for the National Institute of 
Education, says: 

"3. Negotiations between the OE/NIE team and a three-man 
FRMS group, headed by (the new FRMS Project nirector), will 
begin on Tuesday, June 5, in Washington. It is exnected 
that the negotiations will continue for several days until 
both teams agree upon a common technical and budgetary 
position. A major point of consideration will be to com- 
plete all negotiations and contract proceedings in suffi- 
cient time to reprogram the $1,740,000 of OE funds still 
not released to FRMS under their present contract, and 
the approximately $500,000 of OE funds that has been 
transferred to the Office of the Secretary for this experi- 
ment." 

Document (73/06/OlB), from the Director, Division of Tech- 
nology Development to the Group Director, Technological Innovation 
in Education and to the Director, Office of Telecommunications, 
says : 

[The FRMS Project Director] "...further asked if the 
total funds contemplated are about $6,000,000 for FY 
1974 and FY 1975. I responded that the funds for the 
remainder of the project, beginning immediately and 
continuing through June 1975, would be substantially 
less than $6,000,000, but I did not discuss a particular 
budget figure. 

"15, It is essential that we arrive at a technical and 
fiscal understanding as soon as possible. This is 
essential if we are to reprogram the FY '73 monies." 

It should be obvious that most of the resnonsibi lity for 
project implementation should lie with the contractor or grantee 
or the project should not take place. Only in cases of extreme 
deviation from the original consensus and plans should the 
funding agency interfere in a directive mannei . In the case 
of the FRMS Project the absence of a pre- funding and pre-hiring 
agreement on goals and scope made possible the continuing 
destructive dialogue between FRMS Planners and funding 
agency personnel. 

"^here was (and is) strong political support for the Project 
in the Rocky Mountain region. Therefore, regardless of the 
difficulties at the federal level, the Project continued to be 
.^funded and "kept alive" somewhat unevenly, until late in 
*?.,973, when somewhat firmer funding guidelines were established. 



General Discussion 



As lonf; as funded activities are meeting agreed-unon goals 
and objectives, implementation of project-based (rather than 
fiscal based) funding might buffer worthy projects from the 
effects of decision making that is affected by political 
considerations and changes. 

While large projects are likely to be cooperative endeavors, 
responsibilities must be well defined. The funding agency will 

usually decide which responsibilit les for major project activities 
will lie with the funding agency nd which will be with the on- 
site project personnel. 

For projects of large scale or scope, decisions about 
operational procedures and personnel should be made by the 
grantee with agreement on major points from the funding agency. 
The funding agency should review operational procedures to 
insure that they reflect sound management practices and will 
be workable in the context of the project. These procedures 
should be included as part of the original grant agreement 
discussed in t e first recommendation. 

When the contractor or grantee does not have the complete 
confidence of the funding agency, the original scope of the 
project can be small, or can be staged with definite evalua- 
tional milestones occurring during early phases. Mutually 
satisfactory changes can then be made in the original grant 
document if there is a desire to revise or enlarge the scope 
of the project. 



Planning and Operational Considerations 



Recommendation 7 ; Overall planning, fund negotiations, 
"needs" surveys, and operationalization of policy should be 
completed by a small team composed of applicant agency 
personnel, ideally with the support and assistance of 
policymakers or funding agency personnel as well. 

Recommendation 8 : All early steps should be complete 
before a large effort is launched, numerous staff hired, or 
large amounts of funding committed. 

Recommendation 9 : Avoid having a commitment tc planninp act 
as a psychological lever, preventing progress that could 
take place, by too rigidly defining which are properly 
••planning" tasks ana which are not. 



Recoininendation 10: Any attempt to incorporate the ideas of 
tne "population as a whole" or of any interest groups ("needs 
surveys," e.g.) should be initiated during the preplanning 
stages of the project. 

Recommendation 11 : As early as possible, research planners 
should be involved with setting oblectives. They should 
create measurement instruments, make certain that procedures, 
samples, and field procedures will meet research, project, 
and evaluation objectives. 

Recommendation 12 : Detailed project-wide objectives con- 
sistent witn grant agreements should be written. Objectives 
««hould be defined for field procedures, hardware and software, 
and management, as well as for programs and fnr instructional, 
or "treatment" variables. 

Recommendation 13 : An attempt should be made to spell out 
programming or instructional, or "treatment" objectives in 
very fine detail, and to develop measurement instruments in 
coordination with setting objectives as one of the first 
steps in planning, to be taken in parallel with such activi- 
ties as milestone charting, budgeting, and initial contacts 
with cooperating or relating agencies* 

Recommendation 14 : Make baseline measures of pre-project 
conditions . Such measures provide information that is too 
important to overlook. 



The FRMS Project 

The FRMS staff grew quickly. Instead of a small initial 
planning team, there was a moderately large, newly hired staff, 
many of whom participated to some extent in preparing the 
planning and proposals documents. This large scale operation 
without well-coordinated leadership probably contributed to 
the lack of a cohesive, well-coordinated plan that would 
indicate cooperation within the entire project to the proposal 
reviewers . 

If goals, plans, and methods are agreed upon early, it 
may be possible to avert the development of within-project 
factions. These, if allowed to proceed unchecked, can pull 
the project apart from within, as well as cause conflicts 
to develop between the project and the funding agency. 
Furthermore, strong internal factions can cause the project 
to have a schizophrenic image, making it nearly impossible 
to deal satisfactorily with persons and agencies with whom 
the project must relate or interface in order to accomplish 
its mission. 







Because of a perceived need to hurry, which contributed 
to the hiring of the large initial staff, the project was caught 
in the dilemma of being both planning and to a certain extent 
operational. One example of this can be seen in the attempt 
to design ''user based programming". The FRMS Demonstration 
expressed commitment to programming based on "user needs" but the 
results of the early planning did not provide either a clear 
theory structure or a firmly data-based definition of what 
"user needs" might be. (72/08/250 says, "...it is difficult 
to understand why they are still » identifying needs of the 
intended audiences' as they go into the actual production 
phase." Document (72/10/20) says, "With regard to lack of 
regional minority group target population planning inputs, 
this submission lacks even the unbacked-up asse'tions 
regarding such which appeared in the previous document." 
Document (72/10/22) from the funding agency states, "There 
Is inadequate indication of purposeful and substantial 
Involvement of user groups in the planning and developmental 
processes and of projected continued involvement." 

The Federation reply (72/10/31) Identified approximately 
10 activities that the FRMS staff considered oart of "needs 
assessment." However, these activities did not match the 
funding agency's idea of what "user needs assessment" might 
be. It Is not the purpose here to pass judgmeni on the 
adequacy of the activity, but rather to say that there was 
a considerable mismatch between the Ideas of the funding 
agency and the operatlonalisation of the project. 

Anpropriate early planning within the framework of the 
first recommendation could have avoided many of these problems. 
The entire goal and objectives setting operation, development 
of instruments, and planning In general depend upon results 
of any legitimate needs survey. 

Many planning problems can be avoided if research and/or 
evaluation planners are Involved with setting Initial goals 
and objectives. They should create measurement Instruments 
and make certain that populations, samples, and field 
DTOcedures will meet research, proiect, and evaluation objec- 
tives. These details of planning help proposal reviewers 
estimate how well the planners know what they are poin^ to 
do. The FRMS had only fragmented, component -based plans for 
data collection. The Stanford University ev.iluatlon planning 
team prepared on Its own a draft data gathering system, in 
order to satisfy their contract provisions. The pwms staff 
did not wish to prepare for data Rathering at that time (January, 
1973). Therefore, at the request of FRMS staff, the Stanford 
Interim Renort on the Documentation System for the FTD was 
labeled "nraft." This "Draft System" was not used by F^^<S or 
referred to thereafter in connection with the Project. 
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The FRMS PTl) Uld not wake early, systematic attempts 
to determine user needs and preferences. Baseline measures 
of pre -Demonstration conditions were not made. Hser needs 
and baseline measurements could have been and should have been 
coordinated. Baseline measures can often be made In the process 
of assessing "user nreds," since the status of the projected 
target audience is one kind of data that may be used in deter- 
mining what sort of materials and procedures would most 
suitable. 

Too many of the TTD's early resources were sncnt on hiring 
a large staff. Once the staff was hired, a significant 
proportion of Project efforts were expended in revising 
proposals and replying to funding agency requests end 
challenges. This money could have been better employed for 
data gathering and substantive planning. 



General niscussion 

Specification of detailed objectives early in planning 
can help to avoid situations in which planners state one 
set of goals and ouerationalizers implement quite a different 
set. Room should oe left in the project for legitimate 
revisions of early plans and objectives. Revisions should 
be based on data that point definitely to specific things, 
not on unverified assumptions. Revisions for project improve- 
ment should be considered during regular evaluations, as 
discussed in Recommendation 6. 

It takes a certain amount of time to develop valid and 
reliable measuring instruments. Therefore, instruments 
must be developed in coordination with setting objectives 
as one of the first steps in planning, to be taken in parallel 
with such activities as milestone charting. Baseline measures 
of pre-system conditions are extremely important. No 
amount of rhetoric can substitute for an adequate baseline 
measure when one is trying to compare the new system with the 
old. Making baseline measures is another compelling reason 
to state objectives and develop measuring instruments early 
in the planning period. Tf changes are needed as a result of 
unforseen developments or of natural evolution in the project, 
at least some evaluative data from earlier procedures and 
materials will be available. Mmost every large scale project 
that has been conducted has lacked baseline measures. For 
example, in American Samoa, "...no system-wide measurements 
were made of the amount of learning that took place in the 
Samoan schools before the new educational plan was introduced 
in 1964. .. ," (Nelson, 1970) . 
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Personnel and Project Staffing 



Recommendation IS : Obtain personnel of the highest quality. 
An already-established team can often function more efficiently 
than a newly-made team, and the schedule of the project should 
dictate to a certain <»xtent whether or not it is reasonable 
to start up an entirely new team effort within the time 
available. 

Recommendgition 16 : Obtain a strong project director or strong 
co-directmg team. Strong, cooperative leadership is almost 
certainly essential to success. 

Recommendation 17 ; Obtain personnel (especially those in high 
level or otherwise critical positions) who will commit them- 
selves to serve overall project goals, not to develop within- 
project "factions", and to forsake personal commitments to 
"pet ideas," until project goals have been achieved. 

Recommendation 18 ; Especially in developing nations and 
regions. It win usually better serve the nationalistic 
purposes, as well as the more altruistic purposes of 
upgrading the national or regional capability, to choose 
as project director a professional from among the citizens 
of that region or country. If the on-site officials or 
the funding agency feel that the region is lacking in 
qualified professionals, possibly a co-directorship could 
be arranged with the best qualified native professional 
being supported by a qualitied professional from a different 
region or nationality. 



The FRMS Project 

The FRMS Project relied almost exclusively on newly-hired 
staff, although personal contacts were extensively relied upon 
for staffing. Many- -through not all- -staff in the PCS subcontract 
for content in Early Childhood Development had worked together 
before or were acquainted. This made it somewhat easier for 
that component to begin operations and to function as a team. 
Perhaps this established team feeling also contributed to the 
building of separate within-Component experiments, rather than 
to cooperation with the other, less-well-acquainted Project 
staff members. 

In July of 1972, an Assistant Project Director was hired 
who was expected to coordinate the activities and make the project 
sail smoothly. The task was difficult in the situation where 
the independent operational styles and individual "component 
director" scheme was already well established. Antony .lay (1967) 
makes comments that appear pertinent to the situation: 



"What do you do if you inherit a corporation or 
division or department which is in a state of baronial 
war? ...The trouble is that you are likely to be on trial 
to some extent yourself, and if the barons who are 
divided in everything else are united on the fact that 
vou are a menace, you are not likely to last long enough 
to realize the fruits of your f irmness . . . . In the early 
stages each baron is anxious to try out his strength, 
and some may be strong enough to think they could emerge 
as top baron and become king. Some even do, which creates 
quite a different situation." 

In any case, (72/12/18) comments on some evidence of the 
remaining "separatism": 

"The assumptions which accompanied your budget 
submission of November 17, 1972, stated that approxi- 
mately 180 hours of programming will be used during 
the demonstration. In our discussion with the staffs 
of the Early Childhood and Career Development components, 
their estimates for upduplicated programming add to 250- 
300 hours. Career Development spoke of about 160 hours 
of programming, while Karly Childhood indicated that they 
would use more than 100 hours of new programming. This 
inconsistency must be clarified." 

Document (73/02/26A) states: 

"I still don't know how 'utilization' relates to you 
[Career Development Component], or the Farly Childhood 
Development Task Force. I read the information you so 
kindly gave me and came away feeling that the entire 
management system was burdensome. I hope I'm wrong." 

Earlv in 1973, the funding agency began to request that 
a full-time Project Director be hired (73/01/08; 73/02/27). 



It was not possible to appoint the Assistant Project 
Director to the post of Project Director because of a notential 
conflict of interest and because his professional skills were 
not in television and broadcasting. Tn May 19''3 the Director 
of the Broadcasting and Engineering Component was appointed 
Project Director, and the project was almost comnletelv re- 
structured. It was decided to retain only the Career Develop- 
ment content; content personnel in that component would become 
pirt of the Production staff. Research and data processing 
would be divided and both would he internal in the project. 

Memo ("3/05/05) from the ITD Project officer and ro-Proiect 
Officer to the Associate Commissioner for Tducational Technology 
indicates the directions for the changes: 
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"1. Although this analysis indicates that the Broadcart 
and Fngineering component of FRMS would be supported 
bv a separate contract from HEW/OS, as is currently 
the case, we see no need for this in future. We 
suggest that Broadcast and Engineering be brought 
under a single contract to FRMS from OE/NIF.. This 
would provide better coordination and management 
control by the FRMS Project Director—both internal 
coordination and management control have been serious 
FRMS shortcomings in the past. Any services that 
FRMS would provide to support the Alaska, Appalachia 
or health experiments would be arranged by direct 
contact between the appropriate groups and FRMS. 

"2. OE/NIF: should make a decision about the future of 
the project as soon as possible. Tf our recommendations 
are followed, the project staff would have to be reduced 
from its current size of about 70, to about 23 people 
at FRMS Headquarters. These people should be given as 
much time as possible to find new positions, particu- 
larly since some of them might be interested in academic 
posts . 

"3. As soon as a decision is reached, FRMS should be 
required to develop a new set of plans, organizations, 
and budgets covering the period from the present to 
the end of the project in FY 1975. Until this is 
done, no additional funds should be released to FRMS, 
and we recommend that they be instructed to restrict 
current expenditures to a minimum." 

As one outcome of the reorganization following the NIE 
site visit in April 1973, a three man team was appointed at the 
federal level to negotiate changes in funding for the Demon- 
stration. This team included the Director of Telecommunications 
Policy at HEW, the group director of the Technological Innovations 
in Education for the National Science Foundation on assignment to 
the National Institute of Education, and the National Institute 
of Education staff member who had been the Demonstration's co- 
monitor since 1972. At the regional level, the Governor of New 
Mexico and Chairman of the Board of Directors of the Federation 
appointed a three-man executive committee for the project con- 
sisting of the President of the FRMS, the Governor of Idaho, 
and the Superintendent of Public Instruction in New Mexico. 



General Discussion 

As was emphasized earlier, a project has a better 
chance of success if it has a strong director, competent staff, 
and involvement with both the local professional and lay popu- 
1 at ions . 



76 



The ETVproJect in Colombia (Comstcck % Mtccoby, 1966; 
Com$tock, Maccoby Comstock, 1966) had as nirector a orofessional 
from the U.S., employed by the Peace Corps. He was responsible, 
along with other Peace Corps personnel, professionals from the 
United States, and some officials of the Colombian government 
and educational system, for developing the original goals. 
Ideas, and plans. Such a director was an Important factor In 
eliminating the thematic problem of translating policy Into 
operational procedures* However, this arrangement was not 
without flaws, since it conflicted with Colombian and Peace 
Corps goals of having the project be as Colombian as possible* 

The reports from the KTV project in Colombia note that 
utilization (field services) was a critical function for the 
project. In this context It Is Interesting that at the beginning 
of the project In 1964 there were no Colombians Involved In 
the utilization function. Those workers were all Peace Corps 
volunteers. By the end of the Peace Corps' involvement in the 
project in 1966 there were thirteen Colombians performing 
utilizaticn functions. 

A lack of Colombians In the early stages of the VTV 
project Is not a direct reflection of paternalism from the 
Peace Corps; rather It Is an Index of the lack of commitment 
of the Colombian government as a whole to the HTV project 
at the beginning of its operation. The report says, 

"At the top levels talented and skilled people are 
doubtful about risking their careers — either by decisive 
support or acceptance of active roles In the new 
undertaking. Major agencies of government give no 
more than tacit support. The project is left on its own 
until it proves itself. At lower levels people wonder 
about the permanence of any jobs offered, and give 
preference to alternatives." (p. 54) 

Such hopes and fears affect any innovative project. 
Policies of year-to->'ear fiscal funding further aggravate 
the problems of obtaining high-quality professionals. 
* Appeals must be made to the pioneering and risk-takinjr 

instincts of well-established professionals, and if the job 
maiket is very free it may be necessary to offer slichtlv 
higher salaries In order to induce top-gvadc people to 
become project staff members. 

To some extent the decision of the Colombian t!Overnment 
to delay any major commitment to the FTV p?-oject forced the 
planners and policy makers to be their own early operat ional i zers . 
Thus the persons performing the critical utilization function 
during the formative stages of the project were more closely 
connected to the policy level than were the rolomhians who 
later supplanted the volunteers. Such close connections 
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between policy makers and itnpleinenters arc not usual in large 
projects, and whiU serving ooth functions increases the burdens 
on the policy makers at this stage, it virtually eliminates the 
problems of garbled communication between the policy maker and 
the implementing personnel that occurs when different people 
are working at different levels at the same stage of the 
project. 

Regardless of the management cooperational structure of^ 
a project, an a ready-established team can often function more 
efficiintly than a newly-made team. Consideration should be 
given to that efficiency when a new project is being contemplated, 
since staffing and training of a new team may be necessary. 
The schedule of the project should dictate to a certain extent 
whether or not it is reasonable to start up an entirely new 
team effort within the time available. 

A top-quality supporting staff is needed to imnlenent 
the ideas spelled out in a planning consensus* Failures to 
meet milestone objectives can occur as a result of either a de- 
ficiency in the conception of the project, a deficiency in 
the operationalization of the plans, or a deficiency in the 
personnel or subcontrtctors that have been selected to carry 
out the project. Fmpirical experience should tell which is 
the case- -or if the problems are a combination of more than 
one. When something innovative is being undertaken, it is 
quite likely that personnel will not have all the necessary 
skills. In this case the overall quality and willingness of 
the person, along with the extent of supportiveness of other 
project and related personnel, determines whether or not he 
will be able to att&in the necessary new skills during the 
project . 



Formative Processes: 
Improving the Project and Its Products 



Allowing for Change 

R ecommendation 19 : Permit the project to be formative 
nature to tne extent that improvements may he made along 
che way, as a result of experience. 



The FRMS Project 

Some FRMS planners felt if they made concrete nlans too 
early, the funding agency might hold them to those plans, 
when it nifht be discovered later that a different nrocedure 
or task would be more functional. 
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If there had been ft formalized original Agreement, any 
necessary changes in the project could have been structured 
around that original consensus and be related to mutually 
agreed upon major milestones and deadlines. These milestones 
can then be evaluated, even though there has been some 
evolution during the project's course. 

General Discussion 

In an innovative project, the best cf all possible 
initial thinking and planning may not be good enough to 
carry the project to completion. Clock (1961) states: 

"...One requirement for effective utilization of research 
is thot the sponsoring organization have the flexibility 
to break with tradition and innovate new ideas. The 
action implications of social research are often radical 
in nature and no matter how sound they may be, most 
sponsors lack the courage to divorce themselves from 
traditional ways of doing things in favor of new and 
what often seem to be untested approaches. The factors 
which make for organizational flexibility in this regard 
are difficult to discern." 

In an innovative project, experience may not be avail- 
able to guide planning. Therefore, it is especially impor- 
tant to provide for change to take place through empirical 
processes . 

Some research and management professionals call such a 
Tormative model the "action- research" model (French and Bell, 
1973). Sashkin et al (1973) conclude "...that the action- 
research model provides the best basis for the effective attain- 
ment of adaptive change in social systems .. .because it seems 
to afford the greatest probability of adding to our knowledge 
about the change process and problems of change." These 
authors describe the action-research model as ",, .primarily a 
process model, in which. . .Data gathering, analysis, and 
diagnosis (research phases) lead to action planning a.id action 
implementation (action), the results of which are carefully 
evaluated (research). This evaluation provides data for 
further diagnosis and action. Thus a continuous cycle of 
research and action provides a general model for problem 
solving and change." 

l^ether an action model or a more formal model is used in 
experiments or demonstrations, any large-scale or long -duration 
effort needs provisions for flexibility. Only by such provision 
can the project take advantage of it^ accumulated experience. 

Recommendation 20 ; Tf the project will involve communications 
(instruf-tion, information, etc.) and communications media of any 
kind, to the extent possible, arrangements for kind, quality, 
management, ani.' facilities for production au^l <1 i sseni nat ion 
should be spelled out as part of the original nlanninp. con^icnsus. 
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The rRMS Project 

The r^MS Project had ma»y delays associated witn produc- 
tion, and a f^reat deal of delay and trouble with production 
nlanning and staffing. The changes in the general fortunes of 
the FRMS Project were accompanied by many variations on the 
Question of production. The issues might be outlined as 
follows : 

1. Would there be new production or would existing programs 

be used? What quantity of each? 

2. Was new production to be accomplished in-house or sub- 
contracted? If subcontracted, would it be within the ^?ocky 
Mountain region or in some, possibly more sophisticated faci- 
lity outside the region? 

3. If any programming was to be developed by or originated 
live by PRMS staff, what kind of facility would be needed? 
Could the facility (ies) be found somewhere within the region? 

4. If new studio facilities would be created, who would 
manage and staff them during the course of che Project? IVho 
would own them after the Project was over? 

5. Would the major un-link to the satellite be located in 
Denver? Mow would that affect the amount of equipment and kind 
of facility that might be needed for in-house production of 
programming? 

The FRMS planners wanted to have as much control as possible 
over programming and facilities. Some ''RMS staff members 
wished to do all programming in-house in newly-developed faci- 
lities, while others wished to subcontract with regional pro- 
ducers. Some wished for a combination of both. On the federal 
side, because of the large number of hours that PRMS aspired 
to produce, and because of negative recommendations made hv 
federal 'consultants about potential FRMS production capabilities, 
the funding agency personnel were equivocal about what should 
be done and what should be funded. Several documents have 
rreviously been quoted that refer to the controversy over 
nroduction, see, for example quotes from (72/07/07) n. 64 
this report, and (73/04/191)) p. 53, this report. 

neneral DiscusMon 

Problems and questions about nroduction and facilities 
arc unique to each situation. If the arrangements can he 
snelled out and a^^reed upon early in the project, it is possible 
that any number of patterns of wi thin-house/oi;t -of -house nro- 
'luction, staffing, and facilitie'^ can be irade to worV. Ml 
such arranpenents dencnd, of cr irse, on the compe*:enco of staff 
and the adqueacv of facilities, counled with realistic tine 
estir.ates for materials to be produced. 
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In the Samoa project the early nhascs promised much success, 
''ho now Governor, Urx loo,, was determined to overhaul the educa* 
tional svstem of the island and provide a new instructional system, 
based on 'TV, that would upgrade the skills of both the students 
and the nsident toachin.u staff. Tquipment was Installed, per- 
sonnel who wore rcsi;?tant to the use of television were ''re- 
lieve*! of their duties, and their contracts were not renewed." 
CSchiamm ct al, 11)67). The project had some difficulties with 
creating a library oi programs that could be used from year to 
year. Two il i f i" i cii 1 1 i os wtM'o fomul in actual practice: 1") the 
tcachinn and filming: staff learned a considerable amount 
from each new experience; the result was that early efforts 
wore not up to the critical standards of later efforts; 
and 1) some of the cont-nt tended to become out of date 
over the vear's tine (H , .rnor I.oe, personal communication, 
hilv 19''2). It is interesting and consistent that when 
Hovornor l.ee loft Anoricnn Snnoa to become a member of the 
n.S. ledoral f'ommun i cat i ons Commission, the educational 
project boqan to oxperioncc many problems connected with 
nismatrhos between nporat ional i nation and the original 
conception of the project. 

*!a t e r 1 a 1 s 

Rcconncndat ion 21 : Materials should be tried-out, revised, and 

refinofl so that they can meet some reasonable objectives before 
pass distribution. 

The f-RMS Project now has a small in-house production studio. 
Partiv owinp to the philosonhv of the Project staff and partly 
bccauso of tho short tino schedule, IRMS has not been abl^ to 
nre-rest materials or field procedures to any qreat extent. 

I.nck of Tiro - tos t i lUT, usun 1 1 y accompanied by a lack of com- 
nirr\ont to pro test inp,, afflicts many projects. The team that 
vi<;it<;d ^Vroric^an Snnpa reports that they "...did not observe 
anv ' < * s tenat ic attenpt to refine a television lesson to the 
point whore 1 ts toachine offecMveness became predictable..." 
This flaw, which is present in r.anv innovative projects, can 
interfere with useful research and clear evaluation. 



S chedules 

''U^conn ondat i on ZZ : Adapt '^oth daily and yearlv schociuline 
0 f p roe ran del \ vo r / \<v\\ as of sub j oct - pat tor content to 
suit real needs rath.or than traditional t^oals that liavc been 
i:nporto:! f ron ani^ther educational system. 

•■'f^k.-c: .ncrvi-ir i on > : P.'-^forc nlans nre nado for anv .eiver 
FfoiTn "~o~r~r> a r t i c I n a n t s , rortain that thev understand and 
ai^ree to ar.v real ■;ch*^i'u 1 i n;: cnrstraints. 



SI 



The FRMS Project 

S':heduling was originally a major concern for the FRMS 
FTD Project. There is no universal schedule of hours for 
classes in the Region. Therefore, by normal scheduliirt , 
some classes might be meeting during the broadcast, while 
others would have a class break during the broadcast. 
Agreement would have to be obtained from schools participating 
in the career development programming that those classes 
would be scheduled at the time the satellite was to deliver 
the programs. Farly childhood programming would have similar 
problems in scheduling to meet the needs of care-givers in 
institutional settings. However, many such institutions 
would not have as iruch trouble adapting to care-givers needs 
as would schools with the schedule of classes for caree ' 
programming. In addition, at some point it would be 
necessary to find out which home audience looks at TV at 
what hours. A more serious difficulty with delivering the 
Harly Childhood programming to homes was that a special 
antenna and signal conversion equipment are needed in order 
for a television set to receive and send satellite-mediated 
signals. Caretakers at home would either have to be part of 
an existing cable TV network, receive programs through parti- 
cipating public or private television stations, or have an 
antenna installed on their property. A thirty foot square of 
poured concrete was thought to be necessary for the installa- 
tion at that time. 

}\ost of the difficulties with audience participation 
and nrogram scheduling remained unresolved at the time of 
the visit of the OE/NIE site visit team, 12 April 1973. 
Oocument (73/04/19D) says, for example, ••However, the notion 
of delivering an audience, or having the audience deliver 
itself, up to 25 or 30 miles, at a time of day not of its^ 
own choosin g, to view TV programmTng is a challenge to any 
rs search design, or the claim of meeting social needs." At 
the date of this writing, scheduling, even without the Harly 
Childhood rompcnent scheduling and audience, oroblems , scheduling 
negotiations are still underway. 

C eneral Discussion 

riexibility and ability to make rational departures from 
tradition are particularly important to developing areas of the 
world. In American Samoa, for example, English was taught as 
a functional second language, yet the Samoan setting dis- 
courages its use. The visiting team recommended that the 
srhooi adopt a year-around schedule, with the long vacation 
Conine in the Samoan summer months (Pecember and January) . 
Sop.o of the language programs that were being shown during 
the re.^ular school year could then be shown during the 
lone vacation, "...educators seem to agree that the 
Southern Hemisphere schedule (approximately February through 
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November) is better suited to Samoa than the normal n.S. 
mainland schedule. There is really -no f»ood reason; except 
the convenience and personal habits of Palagi personnel, 
why Satnoan schools need to be on the same calendar as tho^e 
on the mainland." (Nelson, 1970). 

The flexibility recommended in this report assumes com- 
petent planning and staffing. We do not Intend to recommend 
the sort of flexibility that happens with all live, spontaneous 
programming, nor do we recommend "flying by the seat of the 
pants 



Senior Management and Policy Participation 

Rcco.-nmendation 24 ; There is a need for strong sunnort and 
leadersnin from senior policy and administration officials. 

Recommendation 2S : Whether or not their continued involvement 
is possible, it is important for nolicy planners to nrenare 
clear statements of the intellectual foundation for planninp, 
development, and implementation. 

The FRMS Project 

Policy level persons who participate in Planning a proi^ct 
should remain directly involved with the development and imple- 
mentation. The FRMS FTP had periods when federal policy makers 
were closely in contact with the nroiect and others when they 
were not. The fortunes of the project reflect these dif- 
ferent periods. Commencing on 1 July 1972, the authority 
and responsibility for the FRMS KTD became fncrcasinply divided 
between Dlir.W, Office of Telecommunications T>olicy (which retained 
responsibility for broadcasting and engineering), and thv ^'ntional 
Center for rducational Technology (which became responsible for 
the major project activities). Splitting the authority and 
responsibilities for the project in this way had an unfortunate 
effect, since it prevented continued involvement of nn'-W nolicy 
planners who had originaHy been associated with the mnior nro- 
ject activities and thus led to increased confusion about the 
original goals and objectives of the nroiect. Whether or 
not the criticisms of :;he project by the operacing agency were 
justified, the fortUi'.c>: v"^/ the project were lowest durinc the 
periods when policy level people in DflKW were not involve^' 
with the major grant activities. It was, for example, I'.ifficult 
to obtain funds that on-site personnel believed had been conmitted 
to the project, and there were numerous requests to revi'?o 
plans for projected activities. Nearly all federal communications 
bet\*-ecn .Tuly 1972 and June 1973 reflect these requests. 

The Project has strong political hacking from the f^ocl.y 
Vountain region, and a former state Governor vas oricinallv nar*- 
.ine director for the Project. Owing somewhat to these considera- 
tions, che project continued to be funded in snitc of its 
troubles in Washington. 



As .soon as DUKW policy people again became directly in- 
volved with the major grint activities, the fortunes of the 
project began to stabilize t73/06/01A). After that time 
nearly all of the existing project goals were reformulated, 
as well as the scope and purpose. 

General Discussion 

I.eadership 

While sound structures and personnel practices heln assure 
the effective operation of a project, a strong leader nay be 
able to oversow* >ome defects in organization. 

Strong leadership has contributed to the success of 
projects in American Samoa, Thailand, f'l Salvador, and \iger. 
I-ack of stronjii support from the top has contributed many diffi- 
culties to projects in Nigeria, folombia, and elsewhere (Schramm 
et al. 1967), A change in top level personnel adversely 
affected the project in American Samoa. This phenomenon led 
the visiting team that studied the Samoan experiment: (Nelson, 
1970) to conclude that a formal master plan might have made 
the system "...more secure from a changing political situation." 
In El Salvador (Mayo and Mayo, 1971) where an T-TV Project was 
Planned and implemented by the same group of persons, one of the 
early initiators of the project was also the Minister of Fduca- 
tion. The ETV Project in El Salvador relied in large measure 
upon the financial resources and technical assistance rendered 
by USAIT). However, the country also drew heavily upon its own 
resources and insisted upon a strong local control from the 
outset. The Minister of nducation eventually became directly 
involved with the day to day problems and operations of the ETV 
Project. He was invested both with the project's authority 
and its accountability. This involvement resulted in some 
negative side effects: for example, subordinates came to rely 
overmuch on the Minister's judgment. Judging from the report, 
the overall contribution of strong leadership and centralization 
of Planning and implementation eventually brought about a 
successful project. 



Policy Foundation Docunonts 

The foundation documents prepared by nolicv 'ilnnncrs *>y 
grants and contracts should not be merelv roflection«« on or 
reactions to ronorts and proposals; thpv should be clear state- 
ments of nolicv, criteria for execution, and the intellectual 
rationale for funding. If it seems desirable this can he do.ie 
coonerat i vely with on-site planners, which should facilitate 
the establishment of communication patterns. 

The optimum condition in negotiating research contracts 
exists when "...the interaction involves full and detailed 
disci'ssion en these matters. However, this is sineularly dif- 
ficult to achieve (Hock, 1961). "These matters" include; 
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Clarification of the practical objectives of research, establish- 
tnent of a research design suited to the objectives, monitoring 
of the research while it is in progress, identification of the 
applied implications of the findings and, settlement of adminis- 
trative and financial arrangements. 

One practical reason for maintaining a cooperative decision 
making apparatus is that the policy and fiscal planners, govern* 
ment or private, may be unfamiliar with situational constraints 
at each site (Kotz, 1967). Additionally, funding agency staff 
may be ill-equipped to administer innovative programs. A 
joint clarification of the matters listed above ray prevent 
problems . " 

Finally, a clear policy-level intellectual structure is 
important because mismatches between policy, planning, and 
implementation do occur. For example, the evaluation team 
that visited the innovative educational system in American 
Samoa (Nelson, 1970) felt that it was impossible to meet goals 
that had been originally stated for the prograr.i, given the 
current operational system. For example, one of the goals 
stated near the time of beginnini; of the new system in 1964 was 
that students who did not plan to go on to college should be pro- 
vided opportunities in manual skills, functional areas, and 
general knowledge. It was assumed that such opportunities would 
enable those students to contribute to a better economic anH 
social structure, "...yet the program actually put into practice 
made no provision for specific vocational curricula until the 
final two years of high school..." (p. 49) Such mismatches 
between the policy and planning level and the on-site implemen- 
tation are common. The likelihood of mismatches leads again 
to the conclusion that policy makers should lay out means for 
reaching goals in finer detail than they heretofore have been 
accustomed. 



T. .In a 1969 NCKRD report, it is stated that "...particularly 
in respect to major innovations such as differentiated staffing 
or individualized instruction, [a problem arises] because it 
would appear that at pre«;ent we do not possDss the kinds rf 
administrative and professional mechanisms required to diffuse 
and sustain radical research-based reformulations of instruction 
and education." 



Continuini! Pclation^hin* 



Uccomnendat ion 2ft : As related by Recommendation 6, the ftindinj* 
apency should evaluate the progress of the prant or contract 
at recular periods. These evaluation sessions should cover 
four questions: 1) Was the work that was completed directed 
toward meeting the mutually a|»reed-upon Koals and objectives 
of the nroiect? 2) Is the work nrojjrejistn.c! on a reasonable 
time «cale? ^) Are the costs of the prolect remaining within 
the mutually agreed-upon bounds? 5) Is the completed work of 
acceptable quality? 



The rUMS Project 

Project Monitorinp 

We have said many tines before that mucii of the trouble 
the FRMS Prolect had with federal funding npencics stenned 
from the lack of mutually agreed-upon goals and r^hiectives. 
Although there were statiements of Intent from both Proiect and 
federal personnel, there was never an oripinal agreement between 
them about anv single asoect of the Project. Because o^ the 
strong political support for the Proiect in the RocVy Mountain 
region, both federal and Project people assumed that sore kind 
of effort would be funded. Because the interest in the oroioct 
at the federal level was variabJe (some nersonnel nssuninp larce 
funding and large scale, others asstiming moderate funding 
and small scale, and still others changing their minds about 
amount of funding and Project scale as time passed"), dirfercnt 
signals appeared to be sent to Project personnel bv different 
federal personnel about the same or similar issues. 

As a result both of changes in the federal structure 
(o;; projects and personnel moving to \ir,, for examolc) , and 
of mismatches in concept and Implementation between t .o federal 
contract monitoring agency and the on-site FRMS Project 
staff, the federal monitors in the Office of f'<iucation and 
in the National Institute of Fducation began to have an 
incrf»as ingl y antagonistic position vis-a-vis the Proiect. 
Document (73/02/01) from the Proipct office to the Prcsii'ont 
of FK*1S, says, 

"As we discussed with your staff, we belicvp that 
it will bo beneficial to have third-partv professional 
recommendations, before W.V-'l' makes a liccision to nllow 
the Federation to use project funds to create an iu- 
houso production facility. 

'*V.*e will also require vrittpn statements from onch or 
the broadcast educational television stations in thr 
cipht state area that thev cannot either sinplv or in 
conhination nroducc, in a ti.'ielv or ndoquato mnnnor, 
the 104 hours of video materials that vou nro r>l .mn i nc . 



There are at least five P.TV stations in your aren 
that have received grants under NCnT*s 1-ducationnl 
Broadcasting Facilities Program and, according to 
their applications to us, plan either to produce 
programs for the satellite demonstration or to serve 
as a production center for the region. 

"The last item regards the ore-grant audit and cost 
analysis which (an employee] of OE*s Contracts and 

Grants Division, me^.tioned when you visited with us 
on December 19, 1972. Before the audit can begin, 
there is certain information that our Contracts and 
Grants Division requires regarding your FY 1973 
expenditures and their FY 1974 budget projections." 

In the case of the FRMS project, it might have been 
helpful for the federal government to place a monitor or 
representative of the federal funding agency in residence 
with the project staff in Denver, to work as a cooperative 
member of the project. Some attempt to do this is probably 
indicated by the placement on site of an extra-agency funded 
evaluation team, intended to be a functional part of the 
project. The evaluation planning team, however, believed 
their mission to be cooperative with the Project's endeavors, 
not to provide on-site monitoring of the Proiect staff. 
This external position that was supposed to be internally 
helpful caused many disfunct ional relationships to develop. 
Outside parties that become involved in a project probably 
should have a general, overall project interest and respon- 
sibility, and their role should be a part of the earliest 
planning consensus. 

The amount of difficulty in operationalizing the intention 
that the extra-agency funded evaluation team for the I'TD would 
bo cooperative with the project probably indicates that the 
placement of a federal monitor cum protagonist on site might 
not have worked. However, with'such a' placement the power or 
?.uthority structure would have been clear. With the outside 
evaluation planning team, the authority structure was intended 
to remain with the r:TD project. The responsibility for many 
concrete tasks that had not been made part of any formal 
agreement bet\een FRMS and the funding agency was, however, 
vested in the Stanford team. Consequently, the project nersonnel 
were never certain that the evaluation team did not reoresent 
a threat to the project's internal integrity, and the team was 
not fully effective in carrying out its contractual obligations. 



livaluation of Project Reports 

{•arly progress reports -- requested monthly- -became 
confused with responses to federal requests to make changes 
in or explain aspects of the Proposals. Massive documents, 
apnroximately 300 pages each, were submitted from FRMS dated 



25 June and 28 Julv. This activity Involved the entire *tnff, 
ami waji itself the major **?»ropre»s'* between nroffress retiorts 
for a two to three month period. 



Suggestions For Structuring Tontinuing f^elationshins 

There are two main elements in this discussion. The 
first is the role of the orolect monitor and the second is 

the basis and tiroccdures for periodic evaluation. A jreneral 
set of suggestions for structuring continuing relationships 
is given here, including goals and obiectives, time 
schedules, costs, quality or acceptability of products, and 
progress renorts: 

1) The funding agency should set un its nrolects so 
that the monitor can function as a proiect protagonist. The 
sometime ?^"un(ling agency posture of the contract monitor as 
an antagonist is most destructive. If the confidence 

the funding agencv is so low that the nroiect monitor must 
take an antagonistic position, then the project should 
probably be renegotiated and structured so that the funding 
agency can again take a positive line. 

2) As noted in the P«rsonnel_ section of this paper, 
failures to meet milestones' tor goals and obiectives can 
(onitting acts of Hod and war) occur either because of a 
deficiency in the conception of the project, a deficiency 
in the operationalizatlon of the plans, or a deficiency in 
the personnel or subcontractors that have been selected to 
carry out the project, 

3) In the case where the contractor or grantee does 
not have the complete confidence of the funding agencv, then 
the scope of the project itself should be small, or shoultl 
be staged with definite evaluational milestones occurlng . 
during earlv phases. Such projects should usuallv be funded 
on an explicit contract basis (although, of course, there are 
other reasons for funding by contract rather than bv grant). 

4j If the project is of sufficient duration and the 
fundin.p agency begins to develop confidence in the contractors, 
a provision could be made for increasing the size, scone, or 
funding of the project, or for having fewer evaluational 
ni Icstones . 

S) If the funding agencv does have compli.no confidence 
in the contractor or grantee, then there is no annarent 
re.ison that the ultimate responsibility for carrvinp out 
the work should not He with the contractor or prnntee, with 
the funding aecncy in an assisting position. 
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6) In spite of the experience with the f^RfMS, it mny he 
n j{ood Idea in some cases for the funding; apency to nlace a 
representative on-site. This will work if the intention is 
to he cooperative, hut will not work if the wish for an on- 
site nonitor is a reflection of perceived incompetence 
within the project, since it would he impossihle to he trulv 
cooperative. 



General Discussion 

Project Monitor Function 

Project monitors and others who conduct reviews should 
understand the policy hasis for the proiect. They should be 
able to irnnlement the policy and the operational plannine, 
and they shotild hr able to relate with pollcv people within 
the funding apencv, with other apencies, and with the on-site 
project nersonnel. They should he free as nuch as possible 
from nurolv political pressures that mav affect the proiect. 

tvaluational Procedures 

livaluatlon of -a../ project Is necessary; otherwise the 
costs, causes, and effects of what happened will not be known 
and it will not he possible to avoid the had and reproduce 
the good in future settings. 

If the project, whether contract or prant fimiied, is 
to be evaluated in terns of how well its goals were met 
and within what level of funding, it should be evaluated on 
the b;isis of the terms, conditions, and methods of evaluation 
spelled out in the original consensus. 

j'valuation of major milestones should be every month or 
• everv few months, depending upon -the 'clcs i rc-s and convenience 

of ail sides and the real needs of the project, l"' The most 
difficult activity may be assessing the quality of work. 
Milestones should, thierefore, he stated so that product 
quality is part of the evaluation: this may require arbi- 
tration procedures. .^ milestone would, for example, not 
be acceptable simply because an allotted amount of stn*T 
effort or money had been spent. Planning for anv eiycn cask 
or t.isks should Ite accepted as complete only when criteria 
for accomplishment are clear and ipproval or disapproval is 
explicitly pos s ible . 



TT. Tornrtl progress rerorts should not bo required more often 
than every 5 months, lest the activity of preparing nroj^ross 
reports take place of progress. 
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V.ven though funding should be firmly committed, it should 
be clear that if the milestones are not being met, some help 
will have to be obtained until they can be. If the project 
falls very far behind in meeting major milestones, some 
portions of the effort should be terminated and the project 
should be rescoped and renegotiated. Evaluation of major 
milestones does not have to take place as a part of a giant 
written progress report. Except when a product is being 
submitted, progress reports could be brief- -from 3 to 10 
pages, for example. The evaluation should be based on the 
original grant or contract document, and standards for 
evaluation should be clear and consistent for both the 
evaluator and the project staff. 

Research and Evaluation 



Recommendation 27 : ;^ether funded from without or within, 
research and/or evaluation should be part of the original 
planning consensus for the project. Research and evalua- 
tion planners should be an integral part of the earliest 
planning efforts, even if an external summative evaluation 
is planned for the end of the project. Plans for either 
research or evaluation must be made from a position of 
enough authority that the responsibilities can be carried 
out. 

The FRMS Project 

Influences on the I'^valuation Planning Effort 

The extra-agency funded evaluation team from Stanford 
I.'niversity was placed on-site with the ERMS ETD with the 
intention that the team would be a functional part of the 
ETD. This contract was funded through the Office of the 
Secretary of DHEW, but monitored through the Office of 
Program Planning and Evaluation in the Office of Education. 
Although the team completed 'cheir contract obligations, they 
did not accomplish all that they wished to. Part of the 
reason for this is that the funding and the team were perceived 
as external and not part of the project. The evaluation 
planning group was "forced" upon the project and must have 
seemed to be one mort- thing that the federal people ciecided 
to do to the project. This "outside" group was not par*- of 
whatever original understanding there was between the funding 
agency and :he project personnel. Document (7 3/0 5/09A ; 
from the fMrector of the National Institute of Fducation to 
the Secretary of HEW- - narked DRAFT, the only copy available 
to us) says: 
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BEST COPY AVAIUBLE ^ 

"...The role of the Stanford University evaluation 
teari in helping to conduct a meaningful informative 
evaluation has not worked. The Stanford team, because 
its funding is from a separate contract and it has 
no clear support from FRMS management, has had minimal 
effect on the FRMS activity." 

An additional factor that complicated the evaluation planning 
was that the federal agency staff were about to enter into an 
extensive reorganization, in which the Office of Education would 
be responsible for demonstrations and dissemination, while the 
newly-created National Institute for Education would be responsible 
for research and development. Some federal personnel wished co 
retain the budget for the relatively large FRMS proiect withm 
OH. An iLternal OE memo (72/01/17), commenting on an early 
version of an FRMS proposal read, in part, 

"...anyone who reads it could not classify it as 
anything else but an attempt at research. As such, 
it would have to be transferred to NIH along with 
all otl.er R^n functions. Therefore, T presume that 
it needs extensive restructuring on this basis alone... 
In terms of immediate action, I suggest that two or 
three highly experienced research specialists begin 
work immediately with OE and project staff in order 
to outline the possibilities and limitations. Their 
purpose would be to eliminate the "research" features 
of the present proposal and to specify both what can 
logically be demonstrated and what data can reason- 
ably be gathered to evaluate the success of the 
demonstration. " 

This is a clear example of a mismatch in expectations 
between two funding agencies, since the NIH site visit, team 
clearly expected a research plan for an experiment. In 
addition, the above quotation underlies the mismatch between 
the on-site project and the funding agencies in general, as 
evidenced by the fact that the project did, in fact, continue 
to attempt to plan research, in the face of the contract issued 
in Mav 1972 to an "outside" evaluation team, one originating 
purpose of which was to ". . . eliminate the 'research' features. 

Late in 1972 the vmF> V.TTi was transferred from the ^^'^ice 
of Fducation to the National Institute for Fducation. The 
project was automatically changed from being the (supnosedlv) 
"evaluated demonstration" for which plannint^ had gone on for 
nearly the entire calendar year of 1972 to beinr a "research 
project". The site visiting team that was sent to evaluate the 
PTn in mid-April 1973 was understandably confused about the 
nature of the project. The ease with which it was generally 
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FRMS project personnel were never certain that the 
team did not represent a threat to the internal security of 
the project, vis-a-vis the federal funding agency, which was 
(rightly) perceived from time to time as antagonistic to the 
pro j ect . 

4. The original management structure of the project 
was organized along areas of interest of four component 
directors, rather than along functional lines. Considerahle 
investment went into protecting the independence of each 
component. Each content group, for example, wanted to have 
its own field service staff, to plan its own research and 
evaluation, and to be budgeted on a component basis, rather 
than on a basis of shared functions. 

5. The evaluation team was intended to be cooperative 
with the project; however, considerable responsibility was 
vested in the team, without any but persuasive authority 
for having its ideas included. For example, the data- 
collection documentation that was prepared for the nemonstra- 
tion was submitted as "draft," because the FRMS staff refused 
to have anything to do with it. The team was small, and 
there were no funds to engage in field data collection without 
FRMS staff cooperation. Therefore, no baseline measures, 

for example, were made. 

6. A major charge of the outside group was to plan and 
helT) execute formative evaluation of the overall Demonstration, 
including the programs that were to be shown over the satellite 
system, 'some progress was made by the content staff in pre- 
paring measurable program objectives and measurement instru- 
ments, but uncertainties about what content the programs 

were to have and who the audiences were to be, precluded the 
initiation of a thorough formative evaluation or action 
research effort throughout the period of the evaluation 
planning contract. 



General niscussion 

The issue of whether research and evaluation should be 
internal to the project or external has been considered by 
a number of authors. Lippitt (1961) says, "There are 
plenty of successful and unsuccessful examples of both. 
Different problems are created by each situation." He 
does not discuss having the research or evaluation imposed 
on a project from an outside agency, but says that when ^ 
there is a mutually agreed-upon contract the relationshm 
"...seems to result in a more effective influence situation, 
and a motivation by the organization to 'get their money's 
worth' by using the help that is provided." When the research 



supposed that the project could change from demonstration to 
research illustrates a common misunderstanding o-F distinctions 
between research and evaluation.il 

Whatever the causes, at the end of 1972 and early in 1973, 
there was a "too little and too late" effort to convert 
what had become a giant services demonstration, with some 
within-component research efforts, guarded from the inter- 
ference of the outside evaluation team, into a rigorously 
designed research project. This change took place over a 
full year after the project had been funded and its much revised 
proposal conditionally accepted, and in spite of the fact that 
the contract for the evaluation team had been explicitly set 
up not to plan research. Needless to say, the effort to 
convert the Demonstration to research was not successful. 

In the words of one of the NIE site visiting team (73/04/17) 
"There is a confusion among staff as to whether the project 
is primarily a demonstration (as it is officially named) 
and/or an experiment. Perhaps the two are not compatible as 
the project now shapes up." 



Conclusions About the Evaluation Planning Effort 



It is our conclusion that the external position of the 
evaluation team was not successful for several reasons: 

1. The team was imposed on the project from the outside- - 
an outside evaluation planning staff was not part of an original 
consensus about project goals and operations. 

2. The team was caught between funding agency wishes that 
the project not do research (in the beginning), that it would 
do research (at the last) , and FRMS wishes to have a large 
demonstration of a service delivery system that included 
"research" on all aspects of the system. 



TT — The MCHRu '(1961) report mentions a similar difficulty in 
assessing State ard local educational activities as a result 
of an absence of -igreed upon distinctions among research, 
development, experimentation, demonstration, and evaluation. 



or evaluation resources are located inside, "...it is easier 
to be accepted as 'one of us.' But there often seems to 
be a problem of losing status as an expert resource." 

Clock (1961) notes that it probably does not matter 
very much whether research is performed, as an in-house 
activity or is contracted outside; it is most important that 
the policy maker and the research contract monitor under- 
stand one another: "Whatever arrangement is used, however, 
the kind of relationship which is established between the 
policy maker and the research 'monitor' has much to do with 
how useful any research activity turns out to be." 

He also says that the research and evaluation group 
is less apt to compromise on standards and ethics when funded 
outside of the regular project activities: Furthermore, 
"administrative constraints are a constant threat to the 
maintenance of standards and the research organization may 
feel obliged to abandon or compromise its standards in order 
to vvoid administrative difficulty. This may take the form 
of comrromises in the sample design, ignoring mistakes in 
interviews and the like. The research organization's actual 
behavior, of course, is reflected in the quality, and 
ultimately the usefulness of its research." 



Conclus ions 



Frequently in large projects there is conflict between 
the desire to undertake highly visible and potentially ^ 
significant social demonstrations and the need to provide 
hard and systematic evidence on the results. Ftzioni, of 
Columbia University, has written of the difficulty of 
engineering social systems. Unlike experiments in "hard" 
science, social experiments are not generally tested using 
models that provide for a series of expansions from a tight 
original experiment to concept; they are, rather, created 
full blown. 

Thus it is inherently difficult to achieve a workable 
balance among political constraints, operational constraints, 
?ocial constraints, and at the same time achieve the results 
planned or desired for major social demonstrations. The 
recommendations we present here will hopefully be of assistance 
to those planning social demonstration projects in the 
future . 
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Summary List of Recommendations 



Recommendation 1 : Before significant funds are expended, the 
Grantee and the funding agency must agree on the goals and 
objectives of the project, the piocedures to be used, level of 
funding, developmental and impleme-itational procedures, and 
methods for arbitrating differences of opinion that may 
develop during the project. 

R ecommendation 2 : To the extent p0i>sible, the grant or contract 
cTocuments snouia reflect the initial expectations of all 
individual and organizational parties involved in the grant 
negotiations. 

Recommendation 3 : In order to maintain support, project staff 
must be able to 'demonstrate that ideas are clear, that the under- 
taking is financially possible in a fixed time frame, and tl.at 
the results will contribute to the solution of the problem for 
which the project was conceived. 

Recommendation 4 : The grant or contract documents should 
reflect agreement between the policy and operational levels of 
the funding agency, both of which should be invoT^ved in the 
ongoing monitoring and operation of the project. 

Recommendation 5 : Once goals and objectives are agreed upon be- 
tween agency and project, the original grant or contract agree- 
ment should make certain that projects are sustained long enough 
to insure "...that the program as a whole does not suffer from the 
pressures on all discretionary programs to shift foci to leflecv 
the apparent priorities of the moment." (NCERD, 19e9j 

Recommendation 6 : Government sponsored projects (and others 
similarj would ideally be funded on a project basis, based on a 
well articulated, mutually agreed-upon proposal or work statement. 
Major milestone objectives would be set for review at reasonable 
times. Multiyear projects could operate under these criteria, 
with an overall general project review being scheduled to 
take place once e?.ch year. 

Recommendation 7 : Overall planning, fund negotiations, 
"needs" surveys, and operationalizat ion of policy should be 
•completed by a smf»ll team composed of applicant agency 
personnel, ideally with the support and assistanc5 of 
policymakers or funding agency personnel as well. 

Rocommendation 8 : All early steps should be complete 
before a large effort is launched, numerous staff hired, or 
large amounts of funding committed. 



Recommendation 8 ; All early stens should be complete 
before a large effort is launched, numerous staff hired, or 
large amounts of funding committed. 

Recommendation 9 ; Avoid having a commitment to planning act 
as a psychological lever, preventing progress that could 
take Place, by too rigidly defining which are properly 
"planning" tasks and which are not. 

^Recommendation 10 : Any attempt to incorporate the ideas of 
the "population as a whole" or of any interest groups ("needs 
surveys," e.g.) should be initiated during the preplanning 
stages of the project. 

Hecommendat ion 1 1 : As early as possible, research nlanners 
should bp. involved with setting objectives. They should 
create measurement instruments, make certain that procedures, 
samples, and field procedures will meet research, project, 
and evaluation objectives. 

Recommendation 12 : Detailed i-)roj ect -wide objectives con- 
sistent with grant agreements should be written. Objectives 
should be defined for field procedures, hardwi^-^e and software, 
and management, as well as for programs and for instructional, 
or "treatment" variables. 

Recommendation 13 : An attempt should be made to spell out 
programming or ins tr.ict ional , or 'treatment" objectives in 
very fine detail, and to develop measurement instruments in 
coordination with setting objectives as one of the first 
steps in Planning, to be taken in parallel with such activi- 
ties as milestone charting, budgeting, and initial contacts 
with cooperating or relating agencies. 

Recommendation 14 : Make baseline measures of nre-nroject 
condi t ions . Such measures provide information that is too 
important to overlook. 

Recommendation 15 : Obtain personnel of the highest (iualit\. 
An already-established team can often function more efficiently 
than a newly-made te-am, and the schedule of the nroiect should 
dictate to a certain extent whether or not it is reasonable 
to sta-it up an entirely new team effort uithin the time 
available . 

Recommendat. on 16 : Obtain a strong project director or strong 
co-direct ing team. Strong, cooperative leadership is almost 
certainly essential to success, 

R econnendat ion 1 7 : Obtain personnel (esnocially those in high 
TereT or otherwise critical position?) who will commit then- 
selves to serve overall project goals, not to develop within- 
proiect "factions", and to forsake personal commitments to 
"pet idc»a^>," until nroject goals have been achieved. 



Recommendation 18 ; Especially in developing nations and 
repion;-, it w^li usually better serve the nationalistic 
purposes, as well as the more altruistic purnoscs of 
unpradinp the national or regional capability, to choose- 
as proiect director i professional from amon? the citizens 
of that retiion or country. If the on-site officials or 
the funding agency feel that the region is lacking in 
quail ^■ied professionals, possibly a co-directorship could 
be arranged with the best qualified native professional 
being supported by a qualified professionil from a different 
region or nationality. 

Rjconnendu^ ion 19: Permit the project to be formative in 
nature to tTie t-j-tent that improvements may be made along 
the way, as a result of exi>erience. 

Recommendation 20 : If the project will involve communications 
(instruction , mtormation, etc.) and communications media of any 
kind, to the extent possible, arrangements for kind, quality, 
management, and facilities for production and dissemination 
should be speJ.led out as part of the original planning consensus. 

Recommendation 21 : ^laterials should be tried -out, revised, and 
retineci so that they can meet sorae reasonable objectives bo fore 
mass distribution. 

Recommendation 22 : Adapt both daily and ye.rly scheduling 
of program delivery as well as of subject-Matter content to 
suit real needs rather than traditional goals that ha^'e bc-en 
imported from another educational system. 

Recommendation 23 ; Before rlans are made for any given 
group of participants, be certain that they understand and 
agree to any real scheduling constraints. 

R ecommendati.on 24 ; There is a need for strong support and 
readership from senior policy and administration officials. 

R ecommendation 25 : Whether or not their continued involvement 
IS possible, it is important for policy planners to nrerare 
clear statements of the intellectual foundation for nlanning, 
development, and imf lementat ion. 

Recommendation 26 : As related by Recommendation 6, the funding 
agency should evaluate the progress of the grant or contract 
at regular periods. These evaluation sessions should cover 
four questions: 1) Was the work that was completed directed 
toward meeting the mutually agreed-upon goals and objectives 
of the project? 2) Is the work Progressing on a reasonable 
time scale? 7>) Are the costs of the project remaining within 
the mutually agreed-upon bounds? S) Is the completed work of 
acceptable quality? 



Recomme.idation 27 ; Whether funded from without or vithin, 
research and/OT evaluation sh»:uld be part of the orif.inal 
nlanninp consensus for the project. Research and evalua- 
tion planners should he an integral nart of the earliest 
planning efforts, even if an external sumynative evaluation 
is planned for the end of the project. Plans for either 
research or evaluation must be made from a position of 
enough authority that the responsibilities can be carried 
out . 
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tion Component activities. 

(72/10/16) Director, Utilization Component. Memorandum . 
Concerning State Visit to Idaho. 

(72/10/20) From Staff Member miEW/OE, NCRT. Memo to Acting 
Director DPD, DHEW/OE. Siimmary comments on October 5 FRMS 
submission. 

(72/10/22) From Field Reader. Memorandum . Comments on S 
October 1972 FRMS Proposal. 

(72/10/25B) Stanford Field Team Member (Hall). Notes . 
State Coordinator's Training Meeting. 

(72/10/26A,B,F,F) Stanford Field Team Member (Hall), \otes 
State Coordinator's Training Meeting, nferceived needs assess- 
ment, site selection. 

(72/10/27A) Stanford Field Team Member (Hall). Notes . Poles 
of State Coordinators and State Content Liasons and current 
information policy. 

(72/10/30) Stanford Field Team Member (Hall). Notes . 
General overview of project funding and review ot significant 
current and pending ETD activities. 

(72/10/31) From President, FRMS. Letter . To Acting Associate 
Commissioner NCET, DHEW/OE. 

("2/11/16) Stanford Field Team Member (Hall) . Notes . FTD 
history. 

(72/11/17B) FRMS Management System Staff member. Pnper. 
"Fiemonstration Management System." 

(72/11/20) Stanford Field Team Staff Member (Hall). Notes . 
FRMS F.TD funding. 

(72/11/22) From Assistant Project Director. Memorandum . 
To all FRMS Staff. Director of Production and Director of 
Research have been hired. 

(72/11/26) FRMS Management System Staff member. Paper . 
State coordinator's training meeting. 

(72/11/30) Stanford Field Team Staff f!ember (Hall). Notes . 
Interviexv with FRMS Assistant Project Director. 
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(72/12/04, 04A) Stanford Field Team Staff Member (Hall). 

• Xotes^. Current FTD status. 

(:2/12/04B) FRMS. Nctes. PERT review meeting. 

(72/12/15B) From F.TD Co-Monitor. Memorandum . To FRMS 
Management. 6-7 December FRMS Site Visit: Amplifying 
Addendum to Letter to Proiect Officer. 

(72/12/18) From Director, DTD. Letter . To President 

of FRMS. Discussing changes to be made after 

visit from HTD Project Co-Monitor and Director, DTD. 

(72/12/26) From FRMS Director of Production Component. 
Memo . To Assistant Project Director. Production of program- 
ming material. 

(73/01/08) From Director, DTD. Letter . To President of 
FRMS. Discussing changes in FRMS liii) Project. 

(73/01/19A) Stanford Field Team Member (NMarkle) . Interim 
Report . "General History and Analysis of the Planning nriase 
of the FRMS ETD." To Contract Monitor, DHEW/OE, OPPH. 

(73/01/19B) Stanford Field Team Member (Hall). Interim 
Report . "Historical Record of the Planning Phase of the 

♦ ™f5^TD." To Contract Monitor, DHEW/OE, OPPE. 

(.73/01/26, 26A) FRMS Minutes . Subtask Monitor Meeting. 

(73/02/01) From Director, DTD. Letter . To President, 
Federation of Rocky Mountain States"! Tomments about 
FRMS response to 18 December, 1972 letter from Director, 
DTD. 



(73/02/26A) From Education Program Specialist, Career 
Education Task Force. Letter . To Director Career Develon- 
ment Component, FRMS. Questions about scrinting for produc- 
tion, staff, revised statement of objectives, and analysis of 
early data. 

(73/02/27) From Grants Officer, OE's Contract and Grants 
Division; Director, DTD; Program Specialist, HFW. Letter . 
To President of PRMS, Letter requesting chanp.es in scope 
' of FTD Project. 

(73/03/21A) FRMS Subtask Monitor's Meeting. Notes . F.TD 
s tatus . 
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(73/04/06) From Grants Officer, OE's Contracts and Grants 
.* Division; Director, DTD; '^-'rogram ?5pecialist and Co-Project 

Officer. Letter . To President, FRMS. Letter requesting 
a response" to 27 February letter instead of alternative 
Vlan as suggested in letter from FRMS dated 29 ?larch. 

(73/04/11;) Staff Member, NIE, Site Team Visitor. Report ♦ 
"Review of the Rocky Mountain Satellite Project." 

(73/04/16) Member, NIE Site Tesim. Report . "Report on 
FRMS Site Visit." 

(73/04/17) Member, NIE Site Team. Repor t . "Summary Comments 
FRMS Educational Technology Demonstration"".^' ite Visit, April 11- 
13, Denver, Colorado." 

(73/04/18A) Member, NIB Site Team. Memorandum. To Chairman 
NIE Site Visit Team. Re: Federation of Rocxy Mountain States' 
Education Technology Demonstration. 

(73/04/19A) Member, NIE Site Team. Memorandum . To Chairman 
NIE Site Visit Team. Re: FRMS ETD. 

, (73/04/19B) Member, NIE Site Team. Report . "Review and Evalua- 

tion Site Visit, FRMS ETD, Denver, Colorado. April 11-13, 1973. 

^ (73/04/19D) Chairman NIE Site Visit Team. Memorandum . To 

Director, DTD. Concerning Summary panel review of FRMS ETD. 

(7 3/ 04/ 2 OA) From NIE Team Member. Memorandum . To Chairman 
NIE Site Visit Team. Re: FRMS ETD. 

X73/04/20B) NIE Site Team Member. Report. "Review and 
Evaluation FRMS ETD." 

(73/04/23) From NIE Site Team Member. Memorandum . To 
Chairman, Site Visit Team. Re: Evaluation of F.TD Project, 
FRMS . 

(73/04/24) From Director, Division of Technology Develop- 
ment. Memo . To Associate Commissioner for Educational 
Technology. Restructuring of the Educational Technology 
Demonstration Project. 

(73/04/25) From Assistant in Management FRMS. Memo . To 
FRMS Directors, Associate Directors, J^ubtask Monitors , 
ECS Director of Financial Services, Assistant Project direc- 
tor, and Designer of Management System. Monitors meeting, 
25 April, 1973. 
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(73/05/03) From HTD Project Co-Monitor. Memorandum . 
To Associate Commissioner of F.ducatlonal Technology, 
USOE and Director, OTD. FRM.^ Correspondence file with 
Time Life and NBC Enterprises. 

(73/05/09A) From Director, National Institute of Education. 
Dralrt Me.r.o . To Secretary, PHEW. Restructuring of FUMS TTD. 

(73/05/?l) From Assistant Project Director, FRMS. Memo . 
announcing appointment of full-time Project- Hi rectorHina 
resignation of the Assistant Project Director. 

C73/06/01A) From Director, DTD; through Associate Commissioner 
for Educational Technology. Memorandum . To Dcouty Director 
for the National Institute of Kducatlon. Recent developments 
on renegotiation of FRMS Contract. 

(73/06/OlB) Frjm Director, DTD.. Memorandum. To 

Group Director. Technological Innovation in Education 

and Director, Office of Telecommunications, DHEW. Concerning 

conversation with new FRMS Project Director. 



(73/08/noday) FRMS Satellite Technology FY 1974 Proposal. 
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Meration of Rocky Mountain States, Inc. 



f nU;,AflUNAL 1E.CHN01.0GY OEMONSIh'AIIOfi 



MEMORANDUM 



TO: 
FROM: 

SUBJECT: 
DATE: 



Nancy Hiirke]A^/^J 
Ken Lokey 

Historical Record of the ETD 
August 7, 1973 



Nancy, I have two concerns relative to your request: 

1. You mention that you now have a listing of significant decisions 
and documents. Are these incomplete since a particular date 
and is it from that date to the present a period you would like 
to have additional information about? 

2. If you are not concerned with a particular time period, I am 
afraid your request is not a simple data retrieval task at this 
time. I could possibly assist your contractual effort in this 
case if I knew what your historical record now represents, i.e. 
on an exception basis and primarily for the period of tirfiQ that 
I have been in my present position. 

I will be glad to provide you with assistance if you can clarify the 
scope of your request relative to the concerns above. 



K. L. 
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Federation of Rocky Mountain States, Inc. 




August 16, 1973 

MEMORANDUM 

TO: Nancy Markle 

FROM: Tom Maqlaras 

SUBJECT: Historical account of the ETD 

Listed below are some general Items which may be relevant for your historical 
account of the ETD. As soon as we receive the list of documents which you 
now have on file, we will review It. add any documents that would be of assis- 
tance for th^hlstorlcal record, and delete those which we feel are no longer 
relevant. >^bers of our staff would be happy to discuss any of these Items 
with you for clarification and elaboration. 

1. Communications with personnel In the field. When individuals are 
placed In the field with a dual responsibility to the ETD and to 
the fponsoring agency It Is vital that conmunl cat Ions to these 
Individuals and to the sponsoring agency be through a clearly-defined 
specific channel. In this case, we had a field team that was responsible 
to different components. These Individuals received Instructions from 
various sources at the regional level. This often times led to confusion 
and. in some cases, contradictory Information was relayed to the field. 
Any project contemplating an extensive field organization which Is depen- 
dent on cooperative relationships with a sponsoring agency, should guaran- 
tee firm lines of authority and conmunication with these personnel. In 
addition, relationships with the field should be handled in a consistent 
and well coordinated manner. 



2. The field organization was put in place on the basis of "solid" information 
which later turned out to be Inaccurate, As a consequence, a field 
organization was In place but did not have specific tasks to accomplish 
because of the lack of hard data and firm conrnitments from federal agencies. 
The major point here is that moves into state and local levels should be 
based on firm conmitments from the funding agency. 

3. Project activities in the field were by their very nature lonq-ranqe 
commitments. Unfortunately, the comltments from Washinqton were either 
fuzzy or short-range. As a consequence, the retrenchment that was forced 
upon the project created credibility problems for the entire project In 
the field. 



TO: Nancy Markle 
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We were often In an "emergency Information mode" In that we were 
responding to numerous requests from federal agencies that were 
based on telephone conversations. We often compiled data only to be 
called upon shortly thereafter to submit It In slightly different 
format. Information requests from any source, including federal fund- 
ing agencies, should be received In written format, delineating the 
specific Information requested and the uses for which this Information 
Is needed. 

The project had 'an early and strong commitment to user Input. However, 
the term was not used In a specific context. Our effectiveness In 
working with state and local agencies would have been enhanced had we 
been able to adequately define "user Input," the extent to which we 
Intended to use It, how we would use It, etc. User Input was a wide 
open field with few constraints In Its definition. 

A project of this nature needs a clearly-defined decision making process 
and channel, especially to handle concerns, requests, needs, etc., that 
emanate from the field. The lack of such a mechanism leads to faulty. 
Inadequate, conflicting conmunlcatlon, and loss of confidence In the data 
that Is received by field personnel. This creates a monumental problem 
both for the Internal staff and the field staff. In addition, such a 
decision making mechanism would expedite the work of Individuals and of 
the entire project by establishing expectations and guidelines for the 
activities of all personnel. 

Site selection procedures which were Initially Implemented worked quite 
well because states and local sites were Involved In discussions regarding 
the project. In addition. It provided for the nomination of several sites 
above the number we anticipated to select. This gave field personnel 
choices from which to make their final selection. However, the overall 
site selection process suffered from the lack of a research design which 
was to have furnished the guidelines for the final site selection. More- 
over, subsequent decisions at the federal level dictated that the sites 
would be limited in number, would be non-redundant, and would be rural 
isolated. As a consequence, a well -conceived, effective site selection 
procedure was negated because of these final factors in site selection. 
Also, these changes not only interf erred with the actual selection 
process, but their earlier identification would have led to different 
types of site nominations. 

It is important that activities involving such a project and state and 
local agencies be done with the advice and input of legal counsel. This 
project has wisely adopted this policy, and followed it throughout. 

A basic approach in dealing with our sponsoring agency in each state has 
been one of mutual support and goodwill. This posture has been of benefit 
to the project, especially during the critical phase when the state teams 
were reduced from three to one. 
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10. Personnel actions should Involve a clear understanding of the develop- 
mental nature of the Project and Its Impact on hiring and phasing In 
personnel. When personnel occupy positions, they should assume clearly- 
defined tasks and responsibilities and bring to their position an under- 
standing of the external agencies with whom they must work. 
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